AlexNisnevich.blog

Reading List - 2025

2026-02-08T00:00:00+00:00

Lots of stuff happening in my life right now, so my reading list fell by the wayside this year. Here it is, finally: all the books I read in 2025!

Reviews shorter than usual this year.

★★★★½

The Art of Not Being Governed: An Anarchist History of Upland Southeast Asia – James C. Scott

Scott’s anarchist reinterpretation of Southeast Asian history is densely written but very rewarding, greatly shaping my view of state formation, of zones of state control and lack of control, of Southeast Asia and its mind-boggling cultural diversity. The Art of Not Being Governed is full of fascinating insights, too many to really list in a brief review. I particularly liked the chapters on oral tradition and heterodox religion, and how they could have historically been wielded as tools against state control.

The Bewitched Bourgeois: Fifty Stories – Dino Buzzati [tr. Lawrence Venuti]

A highly readable collection of quirky bite-size stories satirizing Italian society with a dash of magical realism, reading at times like a missing link between Kafka and Calvino. The Bewitched Bourgeois collects stories from throughout Buzzati’s long career, from his earliest works through to some posthumous pieces, but I found the mid-career stories from 1950 to 1960 to be particularly impressive, each one pretty much perfectly crafted. Highlights: “Seven Floors”, “Panic at La Scala”, “The Prohibited Word”, “The Count’s Wife”.

The Day Lasts More than a Hundred Years – Chingiz Aitmatov [tr. F. J. French]

The Kazakh sections of the novel are a real masterpiece, effortlessly connecting the present and past, spiritual and political musings, and some captivating (invented) folklore tales, with a lyricism that comes through even in translation. Unfortunately, the science fiction narrative that is interspersed throughout is clunky and clumsily plotted, and I think the novel would have done better without it. Still, this is a novel I find myself thinking about even a year after reading it.

★★★★

The Sympathizer – Viet Thanh Nguyen

Sometimes these very buzzy novels don’t live up to the hype (admittedly I’m a decade late in reading this one), but I liked The Sympathizer overall, an exhilarating mix of spy thriller and literary reflection on the Vietnamese-American experience. Nguyen is at his best when he lets the novel be a novel: the writing is punchy and lyrical and for the most part the characters are complex and well-written, likeable despite their flaws. The novel really drags in the parts where the author seems more interested in addressing specific grievances than advancing the plot – in particular, I didn’t like the length filmmaking subplot in the middle, with its flat characters that seemed to function more as archetypes than real people.

The Dream Hotel – Leila Lalami

A very timely near-future novel exploring incarceration and surveillance, providing a contemporary spin on Kafka and Kesey. Lalami’s prose has a great flow to it, and I found myself unable to put The Dream Hotel down, even as the plot progressively felt bleaker and bleaker.

The Comic Book History of the Cocktail – David Wondrich and Dean Kotz

Did the world need another David Wondrich cocktail history book? Apparently, yes! The Comic Book History of the Cocktail is perhaps the perfect Wondrich book – it’s much broader in scope than Imbibe! or Punch but more accessible than the Oxford Companion to Spirits and Cocktails, and the graphic-novel treatment actually fits the subject matter perfectly. It’s the perfect length: Wondrich and Kotz do a great job of covering the whole story of cocktails, from the origins of distillation through to the various contemporary cocktail subcultures, while still managing to fit all sorts of colorful details, and the book still never seems to drag. Plus, the recipes are legitimately good.

Istanbul: Memories and the City – Orhan Pamuk [tr. Maureen Freely]

Istanbul is an unusual memoir: moody and dreamlike, dripping with nostalgia, jumping between topics in Turkish history, Istanbul geography and Pamuk’s own life, with shockingly personal recollections. The black-and-white photographs interspersed throughout, most from Pamuk’s own collection, do a fantastic job of setting the scene and often become objects of Pamuk’s musings in their own right, with some interesting art-analysis passages about them. I read this right after our trip to Istanbul but perhaps I would have gotten a little bit more out of it if I’d instead read it prior to the the trip.

Vineland – Thomas Pynchon

I picked up Vineland after hearing it inspired Anderson’s One Battle After Another, only to be surprised to find that the novel and film really have very little in common – not plot or setting or characters, really just a general vibe. Still, Vineland immediate sucked me in with its darkly humorous look at early 1980s America (with more parallels to today’s world than I was expecting). It’s certain more accessible than, say, Gravity’s Rainbow, but that’s not necessary a bad thing – on the contrary, I thought the faster pace of Vineland helped Pynchon articulate his worldview in a clearer way. I thought it was almost a great book, but the abrupt ending was a letdown, with so many unresolved plot threads that it felt like Pynchon just got bored of writing.

★★★½

The Last Soviet Artist – Victoria Lomasko [tr. Bela Shayevich]

The Last Soviet Artist, Lomasko’s second collection to be translated into English, collects her graphic-novel ethnographic reporting from the Caucasus and Central Asia, as well as more personal pieces from the start of the Ukraine war to her being forced to leave Russia. As with her previous collection Other Russias, I found the graphic-novel observational reporting style to be a powerful way to present the lives of marginalized peoples, but I wasn’t a fan of her more philosophical sections speculating about the role of artists in society.

Last Days of the Ottoman Empire – Ryan Gingeras

Gingeras does a good job of making sense of a confusing period – Ottoman Turkey from the end of WWI through the founding of the Turkish Republic – dispelling nationalistic narratives and common misconceptions along the way. I appreciated that he didn’t shy away from discussing darker events of the period or from piercing the mythos around Atatürk.

Memories of the Future – Sigizmund Krzhizhanovsky [tr. Joanne Turnbull]

A collection of speculative-fiction stories (and a novella) that were written in the early days of Soviet rule and never published for obvious reasons. At his best, Krzhizhanovsky has a Bulgakov-like ability to satirize the world around him through magical realism. But I found much of his prose to be murky and unclear – dreamlike, but not in a good way, without a clear sense of narrative progression. I’m not sure how much of this is stylistic and how much is due to the translation. I’ll try to re-read some of these stories in Russian at some point. Highlight: the novella “Memories of the Future”.

★★★

Happy-Go-Lucky – David Sedaris

Sedaris hasn’t lost his sharp wit or his observational eye, but as he’s gotten older and more successful, his writing to me seems to have gotten less relatable and a little bit mean-spirited. I was put off by some stories that seemed an excuse to flaunt his wealth more than anything else, and his Balkan travel story put a bad taste in my mouth.

32 Stories: The Complete Optic Nerve Mini-Comics – Adrian Tomine

32 Stories collects Tomine’s original Optic Nerve mini-comics, originally written between the ages of 16 and 20. It offers an interesting look at very early work from someone still developing his style, and it’s well-presented in a zine-like form factor that mimics the original self-published format of the mini-comics.

Getting Stoned with Savages: A Trip Through the Islands of Fiji and Vanuatu – J. Maarten Troost

Unfortunate title aside, Getting Stoned with Savages is an engaging and very funny travelogue detailing a year that Troost and his wife spent in Vanuatu, along with some time in Fiji. I appreciated at times the irreverent nature of Troost’s writing, but he does have a tendency to play up the more “exotic” elements of Vanuatu life in a way that doesn’t age super well.

The AI Con: How to Fight Big Tech's Hype and Create the Future We Want – Emily Bender and Alex Hanna

Bender and Hanna do a reasonably good job at piercing the bubble of hype around the current LLM boom. I did feel like this book suffered from “preaching to the choir”, in the sense that I personally didn’t get too much out of it as someone already aligned with their position, and I can’t imagine someone on the other side would pick this book up in the first place. I guess that’s unavoidable to some extent.

To Kill a Bird with Two Stones: A Short History of Vanuatu – Jeremy MacClancy

There aren’t many histories of Vanuatu, and I found MacClancy’s work to be reasonably comprehensive, if fairly dry. It was written in the early 80s, so it’s interesting to read about the events of 1980 from such a recent, if perhaps somewhat biased, perspective.

1453: The Holy War for Constantinople – Roger Crowley

Crowley explores the siege of Constantinople at varying time-scales, covering years of leadup to 1453 and then slowing down to an hour-by-hour recreation of the events of the siege itself. I thought that aspects of Crowley’s framing were a little old-fashioned, but I certainly learned a lot about a pivotal event that I’d only known the rough outline of before.

Murder on the Orient Express – Agatha Christie

I know it’s a classic of the mystery genre and it certainly is a fun read, but I just found the ending unsatisfying, for reasons I can’t really express without spoiling the whole thing.

★★½

The Deserters – Mathias Énard [tr. Charlotte Mandell]

Enard intersperses two narratives here – one of a war deserter trying to survive somewhere in southern Europe, and one of a gathering of mathematicians that gets interrupted by the events of September 11th. Unfortunately I found both of these narratives to be a slog to read. I didn’t get much out of this novel.

★½

Lively Isles: A Background to Vanuatu – David Luders

A very “DIY” overview of Vanuatu’s history and culture, written by someone who’s spent a lot of time there but is hardly an expert. The book moves between topics haphazardly, interweaving history and myth in a confusing way that makes it hard to tell what is actually true.

[unrated]

Gathering the Pieces of Days – LeeAnn Pickrell

I’ll leave this one unrated as it was written by a friend. Gathering the Pieces of Days is a poetry collection with a unique premise: Pickrell kept a journal for a year, then adapted the events and musings of each week into a poem. An intriguing glimpse at the little moments that make up life.

Bislama Reference Grammar – Terry Crowley

A readable and well-thought-out grammar of Bislama, the national creole language of Vanuatu. Leaving unrated as it’s unclear how to review a reference work like this.

Reading List - 2024

2025-02-02T00:00:00+00:00

It took me longer than usual to write my reviews this time (for a variety of reasons), but here it is: everything I read in 2024!

★★★★★

Austerlitz – W. G. Sebald (tr. Anthea Bell)

Sebald’s remarkable writing style sparkles even in translation. Although Austerlitz is ultimately a novel about the Holocaust, Sebald approaches this heavy topic on an intimate scale, as we follow the titular Austerlitz’s journey to uncover his identity and his past over decades and across the European continent. Austerlitz is gorgeously written, hard to put down (I ended up reading most of it in a single sitting), and ultimately devastating, with an emotional impact that crept up on me and lasted long after I put down the novel. Also remarkable are the photographs that Sebald intersperses the book with, all archival photographs that he reinterprets to be of and by the fictional Jacques Austerlitz.

★★★★½

Poor Things: Episodes from the Early Life of Archibald McCandless M.D., Scottish Public Health Officer – Alasdair Gray

Reading Poor Things after watching Lanthimos’s film adaptation, I was struck by just how much meatier the novel is – more defiantly Scottish, more proudly working-class, and also more intriguingly structurally, with a neat metafictional layer of framing narration and contradictory accounts that leave the reader bewildered as to what truly happened in the story. Gray seems to delight in these postmodern conceits, but what elevates this novel above so much other postmodern fiction that I’ve read lately is that all of Gray’s literary conceits are in service of a narrative that, for all of its layers of Gothic grotesqueness and Victorian pastiche and anti-capitalist critique, is rooted in something very fundamentally human: our desperate search for meaning in our lives and in the world.

Cahokia Jazz – Francis Spufford

Spufford had already impressed me with Red Plenty, but I was just floored by Cahokia Jazz, which became the novel I recommended to the most people in 2024. Cahokia Jazz takes place in an alternate-history 1920s America where the post-Columbian genocide never occurred, in an Indigenous-controlled state slowly being encroached on by American business interests. The detective plot seamlessly meshes with Spufford’s exploration of racial politics and the development of the main character, a mixed-race jazz-loving detective trying to find his place in the world. Spufford’s painstakingly detailed worldbuilding transports the reader to a unique, doomed world—a feeling I’d previously felt most closely in Mieville’s novels The City & the City and Embassytown and Chabon’s The Yiddish Policemen’s Union. Spufford also demonstrates a clear love of early jazz, and the book is just so perfectly jazzy, in a way that doesn’t feel tacked on but is an integral part of the action. It’s great genre literature, yes, but Cahokia Jazz also shows that great genre literature can be great literature in its own right.

★★★★

Telegraph Avenue – Michael Chabon

What starts as a simple tale of a record store going out of business blossoms into a kaleidoscopic epic that is also a love letter to a very specific micro-region of the Berkeley–Oakland border, circa 2004. Telegraph Avenue is Chabon’s first work set in the Bay Area he lives in, and the attention to detail is so meticulous that I experienced an unexpected bout of nostalgia for a Bay Area past that I came here just a little bit too late to experience. The supporting characters are so vividly written, with such detailed backstories, that at least in one case, I was surprised to find that a specific character was a work of fiction and not an actual obscure jazz musician. And the music – you can practically hear the music from the pages as Chabon draws on what seems like an encyclopedic knowledge of jazz and funk. Like Kavalier & Clay, the novel continues to escalate to practically fantastical levels until coming back down to reality in a poignant, bittersweet ending that masterfully ties up all loose ends.

Because Internet: Understanding the New Rules of Language – Gretchen McCulloch

It’s refreshing to read a perspective on online language use that goes beyond tired clichés (the Internet is destroying our ability to communicate, etc.) and actually addresses what is so interesting about Internet English. McCulloch argues that the Internet provides something new to linguistics: a vast corpus of informal writing that falls closer to spoken language than written language, enabling us to better understand how we communicate with one another in practice. Throughout her examination of Internet English, McCulloch addresses sociolects, typographical tone, memes, and emojis, which she intriguingly argues occupy the same place online that gestures provide in face-to-face conversations.

Ladies of the Rachmaninoff Eyes – Henry van Dyck

An absolute gem of a novella. Van Dyck has a masterfully light touch with the prose, and the dialogue is delightfully witty. Van Dyck’s style evokes the slightly old-fashioned feeling of a comedy of manners, but at the same time, Ladies of the Rachmaninoff Eyes is surprisingly modern for 1965, daring to show the perspective of a closeted gay Black teen with levity and presenting a setting where racial harmony is so typical as to be unremarkable. It’s a delightful comic romp for the first three-quarters, with a powerful dollop of pathos at the end. Kudos to McNally Editions for republishing this out-of-print and forgotten masterpiece.

Croft and Ceilidh: or Corra-chagailte – Colin MacGilp MacDonald

A collection of reminiscences about Scottish Highland life around the turn of the twentieth century, combining personal history, local folklore of the Ross-shire area, and reflections on the past and future of the Highlands. MacDonald wrote Croft and Ceilidh in the 1940s, at a time when the culture he was writing about had already largely disappeared. There’s certainly a tinge of nostalgia throughout the book, but there’s more to it than that. MacDonald is interested not just in describing scenes from Highland life but also in exploring what made this way of life possible and the factors that would eventually make it unsustainable. Some sections of the book do seem to stretch the bounds of credulity, though not to the detriment of the whole work – MacDonald’s disarmingly casual, deliberately hokey style makes it hard to tell which of his anecdotes are from his own life, which ones come from stories others have told him, and which ones are perhaps wholly invented.

SPQR: A History of Ancient Rome – Mary Beard

Beard’s history of Rome combines academic rigor with a breezy writing style that’s a joy to read – not an easy feat. Throughout SPQR, Beard investigates the foundational beliefs that Romans had about themselves, interrogates popular myths about Rome in light of recent discoveries, and illuminates ambiguities and current historical debates, showing that classical history is still very much a living and evolving field. Above all, she shows how the issues that ancient Romans dealt with are not so different from issues we grapple with today. In Beard’s words, “I no longer think […] that we have much to learn directly from the Romans, […] but I am more and more convinced that we have an enormous amount to learn – as much about ourselves as about the past – by engaging with the history of the Romans.”

Malaysian Journey – Rehman Rashid

Malaysian Journey is an unusually structured memoir, with Rehman’s life story interleaved with vivid accounts of a journey he took through each state of Malaysia after a period of exile from the country in the 1980s. Rehman, a prominent journalist who often found himself at odds with the government, makes a point of demonstrating what Malaysia means to him – a multi-ethnic, multilingual society whose diversity is its key strength. The mix of history and memoir is a nice touch, with each providing context for the other, though I got lost in some of the sections on Malaysian politics, with so many acronyms and party factions. Ultimately, the book served as a compelling and poignant introduction to Malaysia, its history and culture, warts and all.

★★★½

The End of Drum-Time – Hanna Pylväinen

Pylväinen’s historical novel of cultural collision between Sámi herders and Nordic settlers in the 1850s is wonderfully written and deftly conjures up the atmosphere of life in the tundras of northern Scandinavia, with fleshed-out characters and evocative landscapes. The portrayal of nineteenth-century Sámi life rings true, and Pylväinen has done her research, spending extensive time staying with reindeer herders in Sápmi. There’s a lot I liked about this novel, and it’s a shame that the plot seemed thin and frankly dull at times, with long stretches of minimal exposition and a bewildering ending that I found not at all satisfying. The End of Drum-Time explores a fascinating time and place, and I eagerly picked it up given my interest in far-Northern Indigenous cultures, but ultimately, the novel didn’t live up to my expectations.

Weird Black Girls: Stories – Elwin Cotman

A collection of off-beat stories, mostly but not entirely in the genre of speculative fiction. Cotman offers a unique and much-needed voice to the genre and shines at melding the mundane with the fantastical in unexpected ways. The one story that departs from this and drops the fantasy entirely, “Triggered,” was also my least favorite – to me, it seemed mean-spirited and tonally odd compared to the rest. I particularly liked “The Switchin’ Tree”, “Tournament Arc”, and the novelette “Weird Black Girls.”

How Music Works – David Byrne

If this is a rockstar memoir, it’s certainly an unconventional and thought-provoking one. Byrne digs deep into how music “happens,” interestingly devoting more space to topics like stage choreography and how musical scenes develop than the art of music itself. I particularly liked the “Creation in Reverse” chapter, which presents his theory of how musical genres develop to fit the physical spaces in which they’re performed, and his timely chapter (introduced in the newer 2017 edition) on “Infinite Choice: The Power of Curation.” However, the book got dragged down somewhat by the final two chapters, which I thought the book would have been better off without – in one, Byrne gets too mystical for my taste about the spiritual properties of music, and in the other, he bashes classical music in a way that feels tasteless to me.

Three Moments of an Explosion: Stories – China Mieville

A wildly imaginative, wildly mixed collection of stories by Mieville. Did I like all of the stories in it? No. Did I understand what was happening in all the stories? Also no. But I was never bored, and at no point could I predict what would happen next in any of the stories – Mieville certainly shows himself to be a master of unexpected twists and turns. I particularly liked “The Dowager of Bees,” “In the Slopes,” “The Buzzard’s Egg,” and “Dreaded Outcome.”

★★★

The Ice Palace – Tarjei Vesaas (tr. Elizabeth Rokkan)

Vesaas has a beautifully lyrical prose style that comes through even in translation from the Nynorsk, and his almost anthropomorphic depiction of the Norwegian landscape is masterful. But the pacing of this novella didn’t work for me – avoiding spoilers, the key action happens close to the beginning, and the rest of the novella follows other characters’ reactions to what happened. I can appreciate Vesaas’s attempt to explore grief, but I didn’t feel I got a good enough view of the main characters for this grief to be plausible.

Spiaking Singlish – Gwee Li Sui

Spiaking Singlish [sic] has a neat conceit – it’s an introduction to Colloquial Singaporean English written entirely in Singlish itself but presented in a way that is more-or-less legible to non-Singlish speakers, thus serving both as an introduction to the topolect and a forceful gesture against the Singaporean government’s “Speak Good English” campaign. I came out of it with a good understanding of the development of Singlish and a great deal of respect for the topolect, though sadly, I didn’t have much chance to practice it over my very short stay in Singapore last year.

★★½

Why Buildings Fall Down: How Structures Fail (2nd ed.) – Matthys Levy, Mario Salvadori

A set of 18 case studies in specific building failures, with a cute story behind it (apparently, when Salvadori showed his mother-in-law his previous book Why Buildings Stand Up, she replied that she’d be more interested in reading about the opposite). As someone who knows nothing about structural engineering, I found it interesting to read the case studies, but the book felt dumbed-down (aside from the appendices, there is no math at all) and disjointed. It felt like I was just presented with a series of disconnected and arbitrarily selected incidents and didn’t come out of it with any mental model of what exactly causes building failures to happen.

★★

Red Team Blues – Cory Doctorow

Red Team Blues is an attempt at a detective novel that, unfortunately, has little in the way of the deduction that makes detective fiction interesting. The dialogue is frankly so stilted that it’s tough to read at times, the characters are one-dimensional, and the plot seems primarily structured to convey Doctorow’s points of view. This was my second attempt to read Doctorow in recent years, and while I agree with Doctorow on the issues he writes about, I can’t stand his didactic style, which seems even worse in his fiction than in his nonfiction.

Kreisleriana – E. T. A. Hoffman (tr. R. Murray Schafer and Max Knight, ed. David Charlton)

Hoffman’s character of Johannes Kreisler, the bipolar conductor with such a mystical love of music that he’s unable to function in normal society, became a Romantic-era icon, inspiring compositions by Schumann, Brahms, and others. Unfortunately, I didn’t find Kreisleriana, a collection of fictional journal entries, mostly music criticism, by the eponymous Kreisler, to be much of an engaging read. The various chapters straddle the bounds of fiction and essay, at times seeming to not differ so much from Hoffman’s own voice (after all, Hoffman himself was a notable music critic). Some chapters rise above the rest, particularly “Report of an Educated Young Man” – a comedic story following the adventures of an educated ape who loves music. But overall, reading Kreisleriana felt like reading the musings of a fictional, very prickly music critic – an interesting concept, but certainly not for everyone. (Note to self: A better introduction to the character of Kreisler would probably have been Hoffman’s later novel Kater Murr, which supposedly has more plot and features a cat).

[unrated]

Mid-Holocene Language Connections between Asia and North America – Michael Fortescue, Edward Vajda

I’ll leave this one unrated, as it doesn’t feel fair to “rate” academic writing. Both Fortescue and Vajda make strong arguments for their respective theses – the Uralo-Siberian hypothesis (of a link between Uralic and Eskimo-Aleut languages) and the Dene-Yeniseian hypothesis (of a link between Na-Dene languages and the small Yeniseian family in Siberia). Still, I didn’t end up entirely convinced by either. Actually, for the Dene-Yeniseian hypothesis, which I’ve loosely followed before, seeing Vajda’s cognate lists laid out made me realize the evidence for the hypothesis is more sketchy than I’d previously assumed. That doesn’t mean it’s necessarily false – it’s just very tough to conclusively demonstrate language relationships at such a distant time depth!

A Belated Wordbots Dev Diary / Retrospective (~1 Year Post-Release)

2024-06-02T00:00:00+00:00

About a year ago, I “released”* Wordbots, the tactical card game with user-created cards that I’d been working on since 2016.

(* I say “released” in quotes because this isn’t really the sort of game that will ever become feature-complete, especially not as a side project. But on April 29, 2023, I declared it fully playable and ready to graduate from alpha into a permanent “beta” stage, in which I’m no longer actively developing new features but continue to fix bugs and make minor improvements as time permits.)

Getting Wordbots from concept to fully playable beta was a journey. It was one of the hardest things I’ve done in my life and by far the largest project I’ve ever started, in all senses of the word – the biggest in scope, the most time-intensive, and the most emotionally draining to work on. Would I have started working on Wordbots if I’d known how long it would take and what a torturous path I’d go on to “finish” it? I’m not sure. But in any case, I’m glad I did it.

I set out to write a brief retrospective covering my experience developing Wordbots, from start to finish. I didn’t expect to write 6,000 words on the topic, so my apologies for the length of this post – I guess I had a lot to say!

(For those new to Wordbots, here’s some information about what is, and here’s more about its technical underpinnings!)

First, a visualization!
A (not so) brief history of Wordbots
What went right
What went wrong
Final thoughts
Acknowledgements

First, a visualization!

Before I talk about my successes and challenges with Wordbots, it might be helpful to throw up a little graph I made illustrating the development process as the story of git commits and releases, from my first-ever parser commit to just after the beta release:

Wordbots commits and releases over time (click to embiggen)

I should mention that I was fortunate to have a lot of help throughout this process, especially from my brother but also from a whole host of people who made contributions of code or art or playtesting or feedback, who are credited here. Still, for various reasons, I ended up making the vast majority of commits for both the core game and the parser, so this chart represents both the development cadence of Wordbots as a whole and my own personal journey working on it.

As the chart makes painfully clear, working on Wordbots was not a totally smooth process. Let’s dig into how it all went down.

A (not so) brief history of Wordbots

The beginning (summer 2016–fall 2017)

The original idea for Wordbots came to me as I was working on open-sourcing Montague, the semantic parsing library that I’d worked on with Joseph Turian and Thomas Kim at the long-forgotten NLP startup UPSHOT.

At its core, Montague is a fancy CKY parser that parses a CCG grammar and maps parsed tokens to semantic definitions, given as lambda-calculus terms in a provided lexicon. It is ancient technology by NLP standards (after all, the CKY parsing algorithm dates back to a 1961 paper), but we came up with a clever design that ties these concepts together in a user-friendly way. In essence, Montague took something that had been possible for decades (domain-constrained semantic parsing) and made it more accessible and (dare I say?) fun to implement.

Montague is a semantic parsing library, based on UPSHOT's English->SQL parser

Our original use case for semantic parsing at UPSHOT was translating English to database queries – hardly riveting stuff. But as we were writing the documentation for Montague, I began to brainstorm other possible applications for it, initially to figure out how to communicate the breadth of what our parser was capable of.

After working through a few toy examples, I started thinking of what semantic parsing could be used for within a gaming context. I thought back to card games like Magic: the Gathering and Fluxx, where individual cards (in Fluxx, sometimes even player-made cards!) could completely alter the game’s rules. I’d also recently played and enjoyed the now-defunct online tactical card game Atomic Brawl and imagined a similar style of card game with cards that players could make themselves, with the card text automatically parsed and translated to a programming language, perhaps JavaScript. While writing the Montague README, I even mentioned this idea as an “exercise for the reader”:

Applications Exercise 5. † Game semantics. Come up with a semantic scheme for representing rule descriptions for a simple card game (think Magic, Hearthstone, etc., but simplify!) For example, a card may say something like “Whenever your opponent loses life, draw a card”. Then write a parser for it.

We open-sourced Montague in March 2016, and Joseph and I gave a talk about it at the Strata conference that June. That fall, I started playing with my game idea, building up a lexicon in Montague that could handle simple actions and trigger expressions like "At the end of each turn, each creature takes 1 damage"` and turn them into JavaScript code. But there was still no actual game that these cards supported – it was just a tech demo with an artificial lexicon vaguely evoking a positional card game.

While visiting my family over Thanksgiving, I showed my brother Jacob what I’d built, and he was clearly as excited about the concept as I was. He had some React experience at the time (I didn’t yet), and we quickly threw together a game prototype in React over the long weekend, using Hannu Kärkkäinen’s react-hex-grid library for the hex grid rendering and logic. By the end of the year, we had a barebones UI that could render a board and cards – nothing resembling a game yet, but enough of a skeleton that we could envision a path forward:

A *very* early Wordbots commit

I left my job at the start of 2017 and decided that I may as well take advantage of my newfound free time. I resolved to stay “funemployed” until the end of the year and try to finish Wordbots by then. I started working on Wordbots full-time in January, gaining proficiency in the React ecosystem along the way, and Jacob and I were able to make rapid progress on the prototype. By early April, we reached our v0.1.0 milestone: a fully functional prototype with working card creation and multiplayer gameplay, albeit limited features aside from that (and it certainly wasn’t much to look at!):

Wordbots v0.1.0 (April 2017)

Progress continued quickly as I built up steam. By the end of April (v0.4.0), we had spectator support, activated abilities, card import/export, turn timer, and a huge amount of new card mechanics. By the end of May (v0.5.4): user accounts, auto-generated documentation, SFX, and more card mechanics. June (v0.6.2): tutorial mode and the “Did You Mean” feature in the card creator. July (v0.7.0): practice mode, in-game animations, and a major UX redesign.

We were quickly running through our initial feature roadmap, and an end seemed in sight. But I couldn’t maintain this pace for long.

Slow but steady improvement (fall 2017–summer 2019)

I’d planned to take all of 2017 off of work, but for various reasons (including a health insurance snafu), I ended up interviewing again in the summer. I started my current job in the fall, and this, combined with a couple of big international trips that I took in quick succession, got me out of the Wordbots groove.

The next few months didn’t bring many major new gameplay features. With my more limited free time, I worked more on ironing out things at the margins: tweaking the UX, fixing bugs, adding new things to the parser, and generally trying to make things more user-friendly.

In April 2018, Jacob and I felt the game was sufficiently polished for an official playtesting round. We’d shown it to a few people before, but never anything like this: we solicited participants in our social networks, ultimately sending out a link as well as some playtesting guidelines to 30 of our friends. And then we waited … And waited … And waited … Of our 30 playtesters, only two gave us substantial feedback, and the vast majority never responded at all. As far as I could tell, even people who really wanted to try Wordbots seemed to have been thoroughly stumped by the experience of trying to “play” it.

Our sad playtest spreadsheet (April 2018)

Needless to say, the failed playtest round was demoralizing for us. Still, it was helpful, if disheartening, to learn that our new-player user experience was, to be frank, garbage, and we began to rethink our priorities. One critical conversation that started from this was our attempt to answer a question that we’d been punting on up until this point:

How could we make playing Wordbots a "fair" experience?

After all, if players can design any cards they want, nothing is stopping a player from making some kind of “Win the game immediately” card and stuffing their deck with it. And this would be totally legal in the only Wordbots gameplay format that existed at the time (we now call this the “Anything Goes” format and discourage people from playing it except with their friends).

We brainstormed a few possible solutions to the fairness question:

Use some kind of ML model to predict how powerful given cards are and assign them in-game energy costs accordingly?
Set up a server constantly simulating games between AI players with various cards to determine empirically how effective each card is in practice?
Set up some kind of market-based mechanic where players could trade cards for in-game currency, where a given card’s market value would determine if it was a “fair” card or not?
Allow players to make whatever cards they want, but establish game formats where each player is equally likely to have access to an overpowered card in-game?

The final approach seemed to be the only feasible one. Based on player feedback (thanks, Adam B!), we developed a format we called Shared-Deck (later renamed Mash-Up), where each player brings their own deck to the game, but both players’ decks are shuffled into one mega-deck that both players draw from throughout the game. The Shared-Deck/Mash-Up format injected new life into Wordbots and made matches exciting again. For perhaps the first time, I actually found myself having fun playing Wordbots with people, not just developing it.

In the fall, thanks to some word-of-mouth among Jacob’s friends, we’d assembled a small team of people interested in working on Wordbots and even started holding regular standups. Unfortunately, it proved hard to divide the work, and Jacob and I still ended up doing the bulk of it. Still, having the regular cadence of standups helped maintain developer interest, at least amongst Jacob and me. And our broader team provided a useful sounding board for trying new ideas and for deciding how to prioritize our work.

Around this time, a succession of particularly hard-to-catch bugs convinced us that it would be worth it to begin migrating the codebase from JavaScript to TypeScript. I was initially leery of such a massive undertaking but was convinced to try TypeScriptifying a little bit at a time, starting with the particularly brittle multiplayer code (and eventually ending, about two years later, with the React components). Little by little, we gained type safety throughout the Wordbots client code. Though it was a slog at times, the TypeScript refactor ultimately paid dividends by eliminating whole classes of bugs and making significant chunks of our code easier to reason about.

JavaScript vs TypeScript LOC in Wordbots, illustrating the pace of our big TypeScript refactor

In April 2019, with v0.12.0 of Wordbots, we introduced the concept of “sets”, another answer to the “making Wordbots fair” question. Sets provided a way for players to curate their own collections of cards and challenge other players to make decks using only those cards. Finally, there was now a way for players to build and use thematic decks of player-made cards without worrying about huge power imbalances.

Wordbots's help text on the Sets page. Note that the Set Draft format wasn't added until 2021.

I intended Sets to be the last major concept added to Wordbots and began to think about what it would take to finally release the game to the wider world. I wrote a document called “Wordbots: The Final Stretch,” spelling out the remaining tasks needed to finish Wordbots, in the four categories of User-Friendliness, Robustness, Community, and Accessibility. Of course, “final stretch” turned out to be a little optimistic.

Distractions and demoralization (summer 2019–summer 2022)

By the summer of 2019, the amount of free time I had to spend on Wordbots plummeted once more as I began to spend every available weekend going to open houses, as part of our quixotic quest to buy a fourplex with our friends to live in – another massive “side project” with huge emotional highs and lows, that perhaps I’ll write about at a later point. In December 2019, we finally closed on a fourplex that ticked off all our boxes but required renovation on a massive scale. And, of course, we know what happened at the start of 2020. We were very fortunate to have the built-in community of our fourplex housemates to weather the lockdown with, but it was still an emotionally draining time. I had the mental bandwidth to manage precisely one project in 2020, and that ended up being the home renovation work, which took the better part of the year and was all-consuming while it was happening.

When I circled my thoughts back to Wordbots in this period, I could no longer picture the end goal; it was just a mountain of work that looked more and more daunting as I spent more time away from it. I began to dread the thought of working on Wordbots, but at the same time, I felt guilty for not working on it. There were times when I couldn’t sleep because I was so angry with myself at “wasting my time”, for putting so much work into this project just to abandon it …

Throughout it all, I did manage some intermittent bursts of motivation to work on Wordbots. I was able to finally finish that massive TypeScript refactor by the end of 2020 – it was a good thing to work on while my motivation was limited because I could think of it almost as a puzzle of getting all the types to line up. Around this time, I also slogged through a significant refactor of how cards are stored in the backend, developing a much more robust system of metadata tracking for cards that fixed a number of bugs and enabled many small quality-of-life features down the line.

During a weeklong “Wordbots retreat” in the Lost Coast (a location strategically chosen for poor network connectivity and few distractions) in the summer of 2021, I powered through the work of creating the Set Draft format, a feature that we’d been discussing as a team for a while, and that quickly became my favorite way to play Wordbots. Drafting provided an elegant solution to the “fairness” issue (since either player was equally likely to draft a given card) while also removing the friction of players having to create a deck before playing. With the Set Draft format implemented in v0.16.0, I could finally see the path forward that had eluded me for the past few years.

Other than that, the major gameplay improvements during this time were all aimed at providing a better UX for new players, including a massive interface redesign, art by Chris Wooten, Help and Community pages, a “New Here?” feature, and more flavor throughout the game, as well as flavor text support for cards.

Wordbots home page before the interface redesign (v0.13.0, July 2019)

Wordbots home page after the interface redesign (v0.18.0, October 2022)

The final push (fall 2022–spring 2023)

I was finally able to get out of my slump and really mount a concerted effort to finish Wordbots in fall of 2022. I’d planned a two-month sabbatical from work to work on another project that fell through at the last minute, so I unexpectedly had a whole bunch of free time, so much free time that after a while I was finally able to push through my mental block and start working on Wordbots in earnest.

In October 2022, I added one final major parser feature – and one that I could only have implemented while on sabbatical because of its colossal complexity – card rewrite effects. Finally, Wordbots had an actual gameplay mechanic that no other card game could match – cards could rewrite the very text of other cards. It’s certainly a silly addition to Wordbots and perhaps not worth the ROI on the effort spent to make it possible. But it makes me smile, and so into the game it went.

I also put a lot of work into a detailed “How It Works” page describing the technical background behind Wordbots. I figured that, even if Wordbots was never going to be a successful multiplayer game with a thriving community (and at this point, I was resigned to this fate), at least it could be an interesting tech demo to a certain group of people like me.

A visual summary of how Wordbots turns text into playable cards (click to embiggen)

Aside: Wordbots as a Symbolic AI island in an LLM world

Around this time, there was, of course, a much more significant development in the NLP world – the November 2022 release of ChatGPT, which changed the public conversation about NLP seemingly overnight. As I grappled with all my feelings about the rise of large language models, one thing I struggled to figure out was, “what is the place of Wordbots in the new NLP world?” After all, as cool as the Wordbots concept seemed to me, it was built on old – let’s say ancient – technology.

I ultimately settled on the idea of Wordbots as a demonstration of the continued relevance of older NLP approaches. As powerful as statistical NLP is today, there’s still value in symbolic AI methods. Could an LLM be used to produce code from card text the way Wordbots does? I’m sure such a thing could be implemented. Still, Wordbots’s symbolic AI underpinnings offer some valuable advantages, chiefly consistency and interpretability. Each term in Wordbots’s lexicon can be used repeatedly in consistent, predictable ways, just like natural language works, and not quite how LLMs tend to work with language. And the Wordbots parser is hardly a black box: every card in the game is happy to show off its full parse tree:

Zoomed-in parse tree from the diagram above (click to embiggen)

This new way of thinking about Wordbots’ purpose helped inspire me during the final push towards beta release and also led me to write the slightly cheeky “No LLMs were used in the making of Wordbots” disclaimer on the About page.

I finally began to work in earnest on checking off boxes on that old “Wordbots: The Final Stretch” doc I’d written back in 2019. Much had happened since then, but the overall roadmap was still solid, and I was able to pick up where I left off. Jacob and I started a weekly tradition of Wordbots playtests with each other, and over the next few months, we found and fixed dozens of issues with our multiplayer implementation. Other playtests helped us identify missing quality-of-life features that we proceeded to add, like support for draws, more robust game disconnection handling, support for card rarities in sets, etc. We load-tested the server to make sure it was up to the task of handling more players. We tried the game in every possible browser (including on tablets). By April 2023, we felt Wordbots had finally reached the point at which it could handle an influx of new players – it could survive in the wild.

And finally, the release! (April 2023 and beyond)

On April 29, 2023, we released Wordbots v0.20.0-beta. We were ready to show Wordbots to the world.

To me, this really meant one thing: posting it on Hacker News. I took a deep breath and made a Show HN post, writing a little about my motivation and process behind Wordbots and providing a link. And then … crickets. The post got a few likes and quickly dropped off the front page.

I was crushed. Seven years of work, and all that anguish, and all I had to show for it was a multiplayer game with no players, that nobody would ever see. And of course, I was mad at myself for having invested so much of my self-worth into a Hacker News post, and for relying so much on external validation rather than just appreciating Wordbots on its own merits. After all, wasn’t the journey the whole point?

I tried posting about Wordbots on some subreddits and my social media, but nobody seemed interested. I resigned myself to the idea that maybe Wordbots just isn’t that cool an idea to people other than me and some of my friends …

One night, a week or so after my initial HN post, on a whim (I might have been a little tipsy) I emailed dang, the Hacker News moderator, asking him if I could please re-submit Wordbots to Hacker News with a more descriptive title that better explained what was so interesting about Wordbots. To my surprise, he responded and encouraged me to re-post it. I did, and this time around the reaction was much more positive, with a constructive discussion and a whole slew of new players joining the Wordbots community and immediately creating their own unique cards.

Wordbots never became a huge phenomenon with a super active user base. And, to be fair, I didn’t expect it to (although it would have been nice!). Not having players constantly online is a bummer because when players do log in, they rarely have anyone to play with. We do have a Discord server that is occasionally active, and some players have used it to schedule games and discuss cards they’re working on.

In the year since Wordbots’s release, players have created 1441 unique cards and published 14 sets. A few die-hard Wordbots fans have gone on to make 100+ of their own cards. So that certainly feels like a small accomplishment!

I’ll admit that I haven’t spent as much time working on Wordbots post-release as I’d have liked. I made one big bug-fix release a few months after beta but lost steam afterward and haven’t done much since. I’m definitely due for at least another round of bug fixes. But remarkably, the game has continued to function even without any intervention on my part – the server is still up, the parser still works, players are still creating cards, and occasionally games are being played, all without critical bugs. I’m pretty proud of what I’ve made, and I’m glad it’s still bringing joy to people.

What went right

1. Sticking through to completion

The single thing that I’m most glad about in the Wordbots development process is that, well … we finished it! There were certainly various degrees of motivation throughout, and for a while there in 2020-2021, it seemed unlikely to me that I would ever be able to push this thing through to completion. Really, if there is one takeaway I want to offer from this massive, massive blog post, it’s “Stick with it!”

2. Investing in developer productivity

From the very beginning, investing in developer productivity was a major priority of mine. Every change to the parser’s lexicon and semantics was accompanied by unit tests verifying that given strings parsed to the right abstract representation, and it really wouldn’t have been possible to keep the parser stable without this, as even small lexical changes could have unexpected impacts on how different sentences parsed (you can check out our current test suite here). And early on in the development of the game client, I built a fairly comprehensive testing harness to make sure cards behaved as expected. (Using Redux for state management actually made this fairly simple – we had one massive reducer that handled all in-game user interactions, and most of our tests just consisted of simulating actions against this reducer and checking the resulting state.)

As our work on Wordbots continued, we made a series of major refactors in 2018–2020 – migrating the entire JavaScript codebase to TypeScript, migrating from Material UI v0 to v1, and completely overhauling how cards were stored in our Firebase database. While these refactors took an immense amount of work in the moment, they greatly improved our confidence in the code and the stability of the whole system, and I’m not sure we would have been able to get Wordbots to beta without them.

3. Soliciting feedback throughout

So much of what makes Wordbots work today – from the game formats that make games fair to all sorts of card behaviors all the way to the look and feel of the interface itself – owe their existence to player feedback. I’m really glad that we continued to playtest Wordbots and continually showed it to new people throughout the development process (even if some of the playtest rounds were demoralizing to Jacob and me due to lack of player interest).

I also just want to mention my immense gratitude to our mega-testers and sounding boards, who gave us the feedback and critique that led to so many features and fixes: Asali, James, Annie, Danny, Adam, Liam, Honza – thank you!

4. Me working throughout the whole stack, from the game client to the parser

This can also be thought of as a con (after all, it’s really a symptom of me not being able to build up a team like I wanted to), but on the bright side, the fact that I had to work on every single part of Wordbots was liberating in the sense that as I lost steam working on one part of the game, I always had the option to shift gears to other parts. This proved very helpful, especially during the lower-motivation periods – oftentimes, my gateway drug for working on Wordbots was fixing parser bugs or adding new vocabulary to the parser, as these were usually such self-contained projects that it was easy to pick up and work on them for short periods of time.

5. The Wordbots concept itself

Finally, I should mention the general Wordbots concept itself as a major “pro” – after all, it was a strong enough idea that the vision was able to push us through to the end despite all the challenges along the way! I can’t think of many projects I’d be able to work on through to completion for seven years, but this has definitely been one of them.

What went wrong

1. Struggling with motivation

In an ideal world, I would have maintained the same level of motivation that I had at the start of the project all the way through to completion. Of course, this isn’t what ended up happening. For a variety of reasons – lack of time, self-doubt, not being sure about what Wordbot’s place in the world was – I struggled with motivation at various times and had a particularly low period in 2020–2022 where I thought of Wordbots as an obligation rather than a fun project to work on.

Could I have done it differently? I could imagine a scenario where I hadn’t run out of steam by the end of 2019 and finished the Wordbots beta then. In that case, we probably wouldn’t have gotten key features like drafting formats and card rewrite mechanics – I hadn’t thought of those yet! But the core of the game would have been released to the world years earlier, with so much less mental anguish on my part. But it’s futile to think about those kinds of hypotheticals.

2. Making a multiplayer game in the first place

While Wordbots’s multiplayer gameplay is undoubtedly a core part of the experience, I do sometimes wish I’d found some way to implement my original “semantic parsing for game mechanics” idea without the multiplayer part. The issue is that multiplayer games require a community of players, and there’s a chicken-and-egg problem in that players aren’t going to keep logging in if there’s nobody to play with, so jumpstarting a community is difficult. While Wordbots does get some activity from time to time, the average visitor struggles to find anyone to play with. I don’t feel as satisfied about Wordbots’s release as I do about my last major game, Untrusted, which, as a self-contained single-player puzzle game, is always playable for anybody who wants to try it.

(The issues with building a multiplayer game also tied into my struggles with motivation, as I slowly realized that there would never be a stable Wordbots player base, causing me to question from time to time what the purpose of Wordbots even was.)

What could a single-player Wordbots have looked like? I’m not entirely sure. At one point I imagined the concept of a Wordbots “puzzle mode” (inspired perhaps by the puzzle mode in Faeria), where the player is required to win the game in one move from predetermined board positions, with the added conceit that certain cards in the player’s hand would have missing words or phrases, and the player could drag and drop from a pool of phrases to fill in the cards in a way that would make the puzzle winnable. I imagined it as kind of a Faeria (tactical card game puzzle) meets Untrusted (fill in the right words to make a level winnable) meets The Incredible Machine (drag and drop tools to complete a challenge) sort of thing. I never attempted to implement this gameplay mode, but it certainly would have been an interesting route to try.

3. Not thinking about the new player experience soon enough

It took us a long time to advance past the proof-of-concept stage, where Jacob and I sought to demonstrate that what we were building was possible from a technical standpoint. Throughout this time, we tended to prioritize new features over UX. And so, for most of our development process, Wordbots just didn’t feel inviting to new players. Even when we set out to run our biggest playtest round in 2018, we’d neglected to think about how bad the user experience was for new players; in the end, most aspiring playtesters didn’t even know what to do when seeing the Wordbots homepage for the first time.

Fortunately, we turned this around and prioritized working on the UX for new players starting around 2020, aided by some helpful alpha testers. But we missed out on a lot of good potential feedback due to Wordbots’s interface being so confusing for the first half of the development process.

4. Never finding a good way to split up work among a team

By the fall of 2018, Jacob and I had assembled a small group of developers who were passionate about the Wordbots concept and interested in helping us. Unfortunately, we never found a way to involve these eager developers in our development process. We’d have regular standups with the whole group, and Jacob and I would try to assign tasks to people, but for the most part, it still ended up being the two of us working on all development tasks (though the rest of the group still offered help in the form of playtesting and discussion of features).

I think part of the problem was Wordbots’ inherent complexity: working on the parser requires pretty specialized knowledge, while the game client is relatively thorny itself due to having to deal with arbitrary card execution as well as all of the already complex systems required to run a multiplayer game. Even tasks that Jacob and I envisioned as simple and self-contained ended up being more complicated than intended, and without easy wins, it was hard for our potential collaborators to have the confidence to approach the rest of the system.

And part of the problem was perhaps that Jacob and I aren’t great at process – we’re both better developers than we are project managers. Setting up a Zube kanban board linked to our GitHub and having occasional freeform standup meetings was about the best we could do, but it wasn’t really enough process to bring new people into the fold.

5. Spending too much time pursuing dead ends

Because Wordbots, at its core, started as a technical demo, it was easy to fall into what I could call a “hackathon mentality” while working on it. I kept falling into the trap of trying to build certain features just because they were technically interesting to work on, even if they were rarely important for the game itself.

Some examples of times that either I or other Wordbots developers got lost in the weeds:

Automated cost calculation – A few of us briefly went down the rabbit hole of trying to have the parsing server predict how much a given card ought to cost by going through its AST, but we quickly realized that this was not remotely feasible.
Random card generation for tests – At one point, I got intrigued by the possibility of generating completely random legal cards for various stress-testing scenarios (for both the parser and the game client). Eventually, I managed some hacky Scala metaprogramming to generate random valid card ASTs within the parser. But for some reason, I got fixated on the idea of generating random well-formed card text, by somehow inverting the parser to turn it into a generator. After spending quite some time perusing the NLP literature in search of prior art on this, I finally gave up.
Server-side card rendering – We have a Discord bot that listens for exported Wordbot card JSON and renders the corresponding card in a readable way. Right now, it just prints a pretty text representation of the card, but for too long, I experimented with various ways to render the card as a PNG on the Wordbots server. The problem is that any technique for rendering React-to-image (and I tried quite a few techniques here) is inherently flaky, and I ran into constant struggles keeping the card-generation service working. Finally, I decided that just rendering cards as text would be sufficient for the Discord bot.

And these are just the rabbit holes that I crawled back out of! Some rabbit holes I managed to get to the end of – card rewrite effects are a good example, or, say, the Scala macro hell that I went through to be able to pretty-print lambda expressions for good-looking parse trees in Montague. So, while I certainly wasted a lot of time on these features, at least for these, I had something to show at the end of it.

6. Sticking to tools we knew

We were able to get off to a running start by leveraging tools we were already familiar with – Scala 2.10 and Montague on the parser side and the JavaScript+React+Redux+Material UI ecosystem on the client side – but this also proved to be a double-edged sword in some cases:

Using JavaScript from the beginning instead of TypeScript led to a painful migration process as we realized how crucial the type-safety guarantees of TypeScript were for what we were doing.
Material UI proved not to be an ideal UI framework for making things look “game-y”, and I had to mess with it intensively over many years to get Wordbots to look more like a game than like, say, a business application.
And my deep reliance on the Scala 2.10 macro system for the parse-tree-visualization feature of Montague makes it impossible to run the Wordbots parser with any version of Scala newer than 2014. Among other issues, this means that the Wordbots parser can never compile down to JavaScript (alas, Scala.js requires Scala 2.11+) for running in the browser.

Final thoughts

I suppose I should write some kind of conclusion, but I don’t know what else I have to add at this point. Working on Wordbots was an enormous undertaking, and while the journey was rocky, I’m glad to have made it to the other side. I learned a lot from this process, and it’s definitely made me a better developer. (And maybe a better project manager, too?)

I don’t know what my next big project will be. I’m still somewhat recovering from the Wordbots process. I’d like to do something a little smaller in scope next, maybe something without an NLP component. But we’ll see!

Acknowledgements

Whew, that was a mouthful of a post! I want to end by once again thanking everyone who helped me make Wordbots a reality, from everyone who contributed code, art, or other additions to the game to all the people around the world who’ve tried making their own cards or playing a round of Wordbots. Thank you all!

Art by Chris Wooten

Reading List - 2023

2024-01-18T00:00:00+00:00

★★★★½

The Remains of the Day – Kazuo Ishiguro

An aging butler taking a road trip in 1950s England seems like a thin premise for a novel, and yet Ishiguro makes it into a profoundly moving work of literature. The bulk of the story is told through the narrator’s recollections as he drives through the English countryside. He begins the journey fully committed to a particular ideal of being a “dignified” butler, which he expounds on in great detail. But by the end of it, cracks have appeared in his belief system – we see just how much damage his unwavering commitment to his ideals has caused to his personal life, just as we learn more about the shortcomings of the lord that all of this butlering was done for. At its heart, this is a novel about the ideas we build our lives around, and about what happens if these towers of ideas that we’ve built up no longer support us.

★★★★

Understanding Comics: The Invisible Art – Scott McCloud

I’ll admit that I haven’t explored the medium of graphic novels a huge amount in my life, but maybe this will change now that I’ve read Understanding Comics. It illustrates the artistic and philosophical principles behind sequential art (iconography, time and motion, the idea of closure, etc), all in the form of a comic. It then goes beyond the mechanics of comics into deeper ideas about the creative process itself. I think everyone involved in any sort of artistic field, irrespective of one’s interest in comics, would benefit from reading Understanding Comics.

Exercises in Style – Raymond Queneau (tr Barbara Wright)

Queneau’s “exercises” themselves – the same short story retold in a hundred different styles, ranging from Haiku to “Mathematical” – and Wright’s classic translation are both an absolute tour-de-force of language. I don’t usually go for this kind of modernist literature, but I couldn’t help smiling at the cleverness of it all. I read a modern edition that additionally compiles Queneau’s previously unpublished exercises and includes new exercises from contemporary writers in such styles as “Beat” and “Cyberpunk.” For the most part, I could take or leave this added material.

Red Plenty – Francis Spufford

Red Plenty is a gripping historical novel about the management of the Soviet economy in the 1960s. No, I’m not kidding. Except it’s not exactly a novel – there’s a mix of fictional characters and real historical figures, with the latter’s dialogue largely lifted directly from primary sources. The narrative chapters are further interspersed with meticulously researched nonfiction chapters on Soviet economics. It’s not an easy topic to make into an engaging narrative, but Spufford achieves the impossible here. He was also clearly passionate about getting all the details right, going so far as to commission his own English translations of primary sources when he realized his lack of Russian knowledge was a barrier. (Even Spufford’s bibliographical footnotes are so inviting and enthusiastic that now I have a dozen more books I want to read about – who’d have guessed – science and economics in the Soviet Union.) Without a doubt, this is the best work of fiction I’ve read about the Soviet Union by a Western writer (sorry, Amor Towles).

The Ladies of Grace Adieu and Other Stories – Susanna Clarke

The Ladies of Grace Adieu is a collection of stories written in the style of 19th-century fairy tales, set in the world of Clarke’s Jonathan Strange & Mr. Norrell. The stories are beautifully crafted in Clarke’s signature style, with a deceptively light tone hiding heavy themes. All in all, it was a delightful read, and I just wish it wasn’t so short. (I also definitely found myself missing the labyrinthine footnotes of Jonathan Strange & Mr. Norrell.)

I particularly liked “Mrs Mabb,” “Mr Simonelli,” and “Tom Brightwind.”

Venice Observed – Mary McCarthy

An absolute masterpiece of travel writing by one of the great essayists of the 20th century. Venice Observed, written over the course of a year that McCarthy spent in Venice in the 1950s, humanizes and contextualizes a city that can too often seem like a cliché. It certainly changed my mind about Venice – after all, it may be a “tourist trap,” but it’s an honest one, having essentially invented the concept of the “tourist trap” in the 17th century. The first half of the book, focusing more on the city’s history, was just one fascinating insight after another. A highlight for me was the essay on how each of Venice’s principal islands essentially functions as a microcosm of Venice in a different time period. The book’s second half veered more into art history, which was still interesting but didn’t grip me as much.

Zeno's Conscience – Italo Svevo (tr Peter Palmieri)

Zeno’s Conscience is a classic of modernist literature that seems to have lost some of its popularity – I’ll admit that I only picked it up because I was in Trieste, where Italo Svevo is something of a local hero. (And I’m glad I did!) The novel is more of a collection of stories, all told through the same framing device of a patient’s diary written for his psychotherapist (a pretty novel concept for 1927!). The patient in question, the titular Zeno, is a mess, but a highly self-aware mess, and he is constantly rationalizing his various compulsions and anxieties, such as his almost daily attempts to quit smoking just because “I believe the taste of a cigarette is more intense when it’s your last.” Zeno, as a character who is both profoundly flawed and tremendously observant, provides a perfect foil through which Svevo can satirize early-20th century Triestine society, as we see Zeno’s misadventures in the worlds of romance and business. In the end, “La vita non è né brutta né bella, ma è originale!” (“Life is neither ugly nor beautiful, but it’s original!”)

★★★½

The Edge of the World: A Cultural History of the North Sea and the Transformation of Europe – Michael Pye

A clever and enjoyable work of pop history about the cultural influence of North Sea civilizations (in particular, Vikings, Frisians, and the Hansa) on Europe from the 7th through the 17th centuries, countering the traditional European narrative of cultural innovations largely stemming from around the Mediterranean. Pye has done his research, and the primary sources he cites provide remarkable glimpses of medieval lives not too different from our own, neatly organized into chapters covering topics ranging from the book trade to “love and capital” to Medieval fashion (an unexpectedly delightful chapter). My one major complaint is that the book often tries to provide simple causal explanations for complex phenomena (I call this the “Jared Diamond style” of pop history). I get why Poe does this – it certainly makes for a “neater” narrative – but it feels insincere and takes away from an otherwise excellent read.

A Very Old Man: Stories – Italo Svevo (tr Frederika Randall)

A collection of Svevo’s last writings, mostly prose fragments that continue the story of Zeno’s Conscience. I’m grateful that these writings have been published and that readers have one last opportunity to enter Svevo’s Trieste. It’s still remarkably witty writing, and Svevo still has his way of making you root for Zeno despite his obvious flaws, but the fragmentary nature of these pieces and the open plot contradictions between them (clearly, Svevo was trying a few different approaches, and at the time of his death, hadn’t fully decided on which path to take) make it a less satisfying read than Zeno’s Conscience.

Revitalizing Endangered Languages: A Practical Guide – ed. Justyna Olko, Julia Sallabank

Revitalizing Endangered Languages is a unique and much-needed addition to the literature on language revitalization – a volume that combines academic rigor with practical advice, sprinkled throughout with real-life examples from revitalization projects worldwide. The scope is comprehensive, covering everything from sociolinguistics to grant writing. Particularly interesting to me were the sections in the beginning on planning revitalization projects and setting goals – much of the existing literature seems to come with pre-existing notions of what the goals of revitalization are, and it was interesting to see this book interrogate these notions. I will say, though, that I was not very impressed by the chapter on revitalization technology, which seemed hopelessly old-fashioned.

Jewish Languages from A to Z – Aaron Rubin and Lily Kahn

I was heartened to see a comprehensive look into Jewish languages - delving far deeper than just Hebrew and Yiddish – aimed at a lay audience. As far as I can tell, this is a condensed and simplified summary of Rubin and Kahn’s earlier edited volume, the Brill Handbook of Jewish Languages, comprising longer passages about languages with a significant attested history (Ladino, Judeo-Arabic, etc.) and shorter sections about poorly-attested Jewish languages. Every chapter illustrates some primary sources, often providing a fascinating look into historical Jewish culture in areas as far removed as South India and the island of Curaçao. Unfortunately, I found some linguistic details here to be simplified to the point of being sloppy and sometimes misleading. I was particularly confused by how the authors conflated distinctive Jewish spoken languages (e.g., Judeo-Tat, the various Judeo-Aramaic varieties, etc.) with languages that just happened to have been written once in Hebrew (e.g., “Judeo-Maltese”), leading to (in my eyes) a far too broad definition of “Jewish languages.”

Smuggler's Cove: Exotic Cocktails, Rum, and the Cult of Tiki – Martin and Rebecca Cate

About as well-written and nicely laid-out a book as can be made about tropical cocktails and the culture around them. In contrast to Jeff Berry’s Sippin’ Safari, which is an excellent work of largely primary research but a little confusingly laid out, Smuggler’s Cove is a very readable synthesis of existing source materials on the history and theory of tropical cocktails and of the Cates’ personal experiences in running an acclaimed tiki bar. The recipes are, of course, top-notch.

★★★

Outwitting History: The Amazing Adventures of a Man Who Rescued a Million Yiddish Books – Aaron Lansky

Outwitting History is an enjoyable and fast-paced read about Lansky’s monumental efforts to preserve Yiddish literature starting in the 1970s. Still, I couldn’t shake the feeling that something was not quite genuine about it. I’m sure that Lansky’s account is generally truthful, but some of the anecdotes had such clichéd, Hollywood-y dialogue (especially in the later sections) that it made me question how many of these events really happened the way the book describes them. By the end, it felt a bit like an advertisement for the National Yiddish Book Center (don’t get me wrong, though – I appreciate their work, and they can certainly use such an advertisement).

The Internet Con: How to Seize the Means of Computation – Cory Doctorow

The Internet Con is a long essay about the virtues of interoperability as a way to curb the power of the tech industry. It was well-written and well-argued, with some clever anecdotes, but it didn’t leave much of an impression overall.

The Island of the Day Before – Umberto Eco (tr William Weaver)

As someone who loved The Name of the Rose and Baudolino, reading The Island of the Day Before was my biggest literary disappointment of the year. Eco set out to write a novel where the universe functions according to 17th-century conceptions of science, and this aspect he accomplishes remarkably well. There’s no denying that it’s a thought-provoking book. The problem is that this may have come at the expense of some of the novel itself. Most of it (spoiler alert) takes place with the main character all alone, and long stretches of the story occur entirely in his head. We get some of the intrigue of The Name of the Rose and some of the colorful exploration of Baudolino, but not nearly enough of either to support the length of this novel. By the end, reading it felt like a chore.

★★½

Users – Colin Winnette

I picked up Users after attending a reading of one chapter of the novel. Unfortunately, the passage I heard was one of the liveliest, best-written parts, and the rest of the book was somewhat of a letdown after that. The premise is timely (it’s all about the unintended consequences of technology), but most of the characters felt more like sketches than fully fleshed-out people, and I was not entirely convinced by the direction the plot went in. I’m still waiting for a great satire of the modern tech industry, but this wasn’t it for me.

The Library of the Villa dei Papiri at Herculaneum – David Sider

A comprehensive, if bone-dry, examination of the history and contents of the charred papyri excavated from the Villa dei Papiri that I read while following the “Vesuvius Challenge.” My biggest takeaway was that far more information has already been decoded from the papyri using traditional methods than I’d expected.

The Final Solution: A Story of Detection – Michael Chabon

The premise – a Sherlock Holmes mystery set during the Holocaust – was intriguing enough to pick it up, but, though elegantly written, it felt far too short and lacking in substance, and certainly not much of a detective story at all. A shame, as I’d really enjoyed Chabon’s other novella, Gentlemen of the Road.

Buildling a DIY Tiki Window for a Home Tiki Bar

2023-09-05T00:00:00+00:00

(Thanks to Annie and Asali for all your help on this project!)

I set out to build a Trader Sam-style “tiki window” for our home tiki bar. You know, this kind of thing, that no home tiki bar seems to be complete without – a screen behind a bamboo façade showing animated tropical imagery:

I perused some Reddit threads and saw a brief but helpful Youtube how-to video but couldn’t really find a complete step-by-step guide, so I ended up largely making things up as I went along.

Here’s how I did it.

Step 1: The Screen

“Digital picture frames” are a thing but all the ones I found online were either too small or too expensive for this project. I didn’t really want to buy a brand-new monitor for this, so instead I went to the legendary Urban Ore and found a decade-old 23” Acer monitor for $20. It was missing its power adapter, but no problem – I managed to find a compatible adapter online.

After testing that the monitor actually worked, I pried off its base and set out to build an enclosure for it.

Step 2: The Housing

The monitor measured about 2” thick at its thickest points, so I figured I could build a reasonable enclosure out of 2x4 boards (for an nominal depth of 4”, so an actual depth of 3½”), giving me enough wiggle room to fit the monitor and any other electronics I would need inside.

I cut the boards to size, double-checked that the monitor would fit (before it was too late!):

and attached them together with pocket screws and Gorilla wood glue:

Once the frame of 2x4s was assembled, I made a “lip” on the inside out of ½”-square dowels, so that the monitor wouldn’t be able to slide forward once placed, and covered those dowels with some rope-shaped trim, just using wood glue for attachment throughout:

I also made a 1¼”-diameter round hole at the center of one of the long sides of the frame, for running a power cord through.

Now it’s time to fit the monitor inside!

The monitor is kept in place with some additional dowel pieces at the back of it (not pictured).

Step 3: The Bamboo

I was fortunate to have some 1¾”-thick bamboo slats from Forever Bamboo lying around from another project. These slats are perfectly sized for this housing – exactly two of them fit side-by-side on each 3½”-thick edge side. On the front, the 1¾”-thick slats don’t quite reach the end of the dowel lip (it’s 1½”+½” thick), but still form a nice effect with the rope trim underneath.

I was a little paranoid about using a nail gun this close to the screen, so we just attached the slats the old-fashioned way, with lots of wood glue. First on the front (using a miter box for the 45° cuts):

And then the sides:

Step 4: The Rest of the Electronics

So now that we have a monitor in an enclosure, how do we want to actually play things on it?

I didn’t really want to put an Internet-connected streaming player inside it (wanted to go a little more low-tech for this), so instead I got a NEUMI Atom 4K digital media player and loaded a flash drive with videos that I obtained using yt-dlp and then Handbrake to convert to H.264. (For what it’s worth, I had to test out a few different HDMI media players and the NEUMI Atom 4K was the only one that was able to handle the 8-hour-long H.264 videos that I threw at it.)

You can kind of see how the back of it all looks here:

It’s really pretty simple – all we have is the monitor itself with its (replacement) power adaptor, the HDMI player with a tiny USB stick inserted, both plugged into a power strip at the top of the box (this is mainly because I liked the aesthetics of having a single cord coming out of this thing – but I also had to bring in a 6” extension cord because otherwise I couldn’t get the akwardly-shaped power brick for the HDMI player to fit within the confines of the box).

Note that I positioned the HDMI player precisely with its IR receiver directly over the hole at the bottom of the enclosure – so that I can control it (change videos, etc) by pointing the remote at the bottom hole while it’s operating.

Oh, what’s that piece of paper doing there underneath the monitor, you may ask? Well, one thing that I didn’t really think about until it was too late is that the power button for the monitor is … at the bottom of it. So the way that I installed it, the monitor was always resting on that button (actually, all of the buttons). Fortunately there was a little bit of wiggle room, so I elevated it just a bit with that sheet of paper – just enough so that the power button isn’t being pressed constantly. And I now have a very janky mechanism for turning the monitor itself on or off – take out the piece of paper and lift and then release the monitor, hitting the power button once. (Fortunately I rarely have to do this, as I have a footswitch connected to the power strip itself that I use to control power to the monitor and media player simultaneously.)

Step 5: Mounting

With the tiki window built, the last thing I had to worry about was mounting it. The whole thing, wood and electronics at all, only weighs something like 20–25 lbs, but I didn’t want to take any chances. I installed a 1x2 cross-bar (you can see it in the photo above) and attached a French cleat rated at 200 lbs to make sure this thing is really going nowhere.

I hung it up in our little tiki nook, and voila!

Coming from essentially zero woodworking experience, this was a bit of an intimidating project to try to take on, but we managed to pull it off pretty much without a hitch. It’s become a centerpiece of our little tiki bar and has definitely inspired me to get more into creative DIY projects – we’ll see where that goes!

"Lingua o Dialetto?" Exploring Perceptions of Regional Italian Languages Over Time

2023-08-05T00:00:00+00:00

See https://nbviewer.org/github/AlexNisnevich/lingua-o-dialetto/blob/main/lingua-o-dialetto.ipynb

Reading List - 2022

2023-01-10T00:00:00+00:00

This was an unusually hectic year, so this set of reviews will be briefer than usual. Flash reviews, if you will.

★★★★½

Independent People – Halldór Laxness [tr. Brad Leithauser]

Perhaps the whole of the human experience is contained within this epic about a small-time farmer’s rise and fall, brimming with wit and tragedy.

Embassytown – China Mieville

The perfect novel about language and language contact, though it’s so much more than just that – one of the most inventive pieces of sci-fi I’ve ever read.

★★★★

Imbibe! [revised edition] – David Wondrich

Wondrich’s meticulous research and engaging writing take what could be a dull subject – 19th-century cocktail recipes – and make it immensely satifying to read about.
Note: I’ve started a project to recreate each drink in the book, as close to the original recipe as possible, though I haven’t gotten very far yet.

Debt: The First 5000 Years – David Graeber

I’m not sure if I am fully convinced of Graeber’s theory of debt preceding money and the constant realignment between the two, but this is a fascinating read regardless, with the earliest sections, on social currencies and “everyday communism”, being the most thought-provoking.

The Shaman's Coat – Anna Reid

I’m a sucker for all things North Asia, so how could I resist Reid’s account of her travels in early 2000s Siberia in search of extant shamanic customs? The history aspect of it is definitely no match for Forsyth’s magisterial A History of the Peoples of Siberia, but the personal travelogue is an engaging read.

How to Do Nothing – Jenny Odell

The anti-self-help self-help book, How to Do Nothing is a manifesto for regaining attention in an age of addictive technology, and manages to be both beautifully written and full of actionable ideas.

The Chukchi Bible – Rytgėv (Yuri Rytkheu) [tr. Ilona Yazhbin Chavasse]

Part mythology, part poetic ethnography, part adventure story, part family lore, The Chukchi Bible is a mysterious and beautifully written ode to a fading culture.

The Dispossessed – Ursula K. Le Guin

I don’t know why it’s taken me this long to read this classic of science fiction, with an extraordinary portrayal of an anarchist “ambiguous utopia”. I just wish the whole thing didn’t unravel with an unsatisfying deus ex machina ending.

★★★½

Why Fish Don't Exist – Lulu Miller

Without spoiling too much, I will say that this is a brilliantly written account of one man’s quixotic scientific journey and the lengths that he would go to to achieve his vision, interspersed with a touching personal memoir.

Sippin' Safari [10th anniversary edition] – Jeff "Beachbum" Berry

What Wondrich does for classic American cocktails, Berry does for tiki. Sippin’ Safari is a fun romp through the world of tropical drinks, but I found the structure of it to be a little unfocused, lacking a clear through-line.

Telluria – Vladimir Sorokin [tr. Max Lawton]

A kaleidoscopic collection of 50 linked micro-stories, all set in a bizarre future of feudal states, human-animal hybrids and super-drugs. I have to give Sorokin credit for his profoundly imaginative world-building, but each chapter being told from a completely new perspective robs it of any sense of real narrative progression.

A Sociolinguistic History of Scotland – Robert McColl Millar

An expertly written work of sociolinguistics that crams centuries of history of usage of Scotland’s two autochtonous languages – Scots and Gaelic – into a slender tome, ultimately striking a critical yet hopeful tone about the prospect of preserving Scotland’s linguistic heritage.

Language and Symbolic Systems – Yuen Ren Chao

Why read a 60s linguistics textbook? In part because Chao, a pioneer of Chinese linguistics, writes in beautifully fluid and conversational way, and in part because this book reflects his own interests more than anything else, and so features incredibly forward-thinking chapters on sociolinguistics, information theory, and even language technology.

★★★

Elements of Clojure – Zachary Tellman

Unfortunately the more Clojure-heavy parts of the book were lost on me (it’s been a while), but I particularly enjoyed the Names chapter, a deep and philosophically interesting exploration of naming in software.

The Truth and Other Stories – Stanislaw Lem [tr. Antonia Lloyd-Jones]

This collection of previously-untranslated stories by Lem was hit-and-miss. Highlights: “Invasion from Aldebaran”, “The Friend”, “The Hammer”, “One Hundred and Thirty-Seven Seconds” (the latter particular timely in the age of GPT-3…) .

The Winter Queen – Boris Akunin [tr. Andrew Bromfield]

A page-turner of a historical mystery-thriller that somehow just doesn’t feel satisfying at the end.

Here I Am – Jonathan Safran Foer

Safran Foer’s trademark wit is still there, but it just didn’t hit me emotionally the way his first two novels did.

The Routledge Handbook of Language Revitalization – ed. Leanne Hinton, Leena Huss and Gerald Roche

What it says on the tin. Part 1 is mostly theoretical essays about topics in language revitalization. Part 2, which I personally found more interestings, consists of myriad case studies from around the world.

Extra! Extra! – «Never Been to a Party» Album Release

2022-12-19T00:00:00+00:00

My dance-rock band Extra! Extra! has just released our debut album, Never Been to a Party, after a year-long recording and mixing process. The album is a blend of new wave, punk, disco, and electronic rock, with influences ranging from LCD Soundsystem to the B-52s to ESG. I think you’ll like it.

The album is available on Spotify …

… as well as Bandcamp and most other streaming platforms.

We will hopefully press it to vinyl next year, if there is demand for it!

Reading List - 2021

2022-01-10T00:00:00+00:00

★★★★½

A History of the Peoples of Siberia: Russia’s North Asian Colony 1581-1990 – James Forsyth
Ah, just the book I needed to feed my obsession with Siberia and The Far East. In fact, it’s so comprehensive and well-researched that I was surprised to even find a book like this in English. Forsyth’s writing can be dry at times but he admirably compiles and presents an overwhelming wealth of information about the indigenous people of North Asia, from the Khanty and Mansi along the Ob river all the way to the Chukchi and Itelmens of the northeast edge. He presents some cultural background, but for the most part this is a history of Russian exploitation, with roughly the first half of the book following the tsarist colonial project and the second half of the book follows the varying rises and falls (but mostly the latter) in the fortunes of the Siberian peoples through the Soviet era. It’s a shame that the story ends in 1990 and the author never put out a new edition, because I haven’t yet been able to find a good follow-up read on contemporary Siberian indigenous history. This is a fascinating and tragic account of a group of peoples that most Western books about Russia barely even mention. [File under History / North Asia.]

Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed – James Scott
Few books have influenced my thinking as much as Seeing Like a State has. Scott’s central thesis is that a diverse range of state activities over the past few centuries can be seen through the lens of legibility - a state, when faced with a complicated, local, difficult-to-understand situation will attempt to standardize and homogenize, sacrificing local knowledge and diversity for the sake of simplification. The introductory example here is 18th century Prussian “scientific forestry”, where natural forests, which formed complete ecosystems but whose timber output was difficult to quantify and predict, were replaced by rows upon rows of identical trees, which were more legible in the sense of being easier to reason about as a system, but of course were much less resilient against weather and pests than the natural, “illegible” forest system that they replaced. From the natural realm, Scott continues to a variety of examples of state simplification of human systems, from Le Corbusier’s high-modernist city planning to Soviet collectivization to Tanzanian “villagization” in the 1960s and 70s, finally closing with a meditation on his idea of “mētis”, the kind of crucial, hyper-specific local knowledge that gets lost during these instances of standardization. Overall, this was an engaging and very illuminating read, marred for me only by a chapter contrasting Lenin and Luxembourg’s visions of revolution that seemed particularly weak / out-of-place here. [File under Political Science or Anthropology.]

East of Eden – John Steinbeck
It’s a classic for a reason. The writing is beautiful (Steinbeck referred to it as his “first novel” after he finished it, and it really does show an incredible maturity of style), but at the same time it feels like it’s missing some of the humanism, the generalized love and compassion for humanity, that is so present in his earlier novels like The Grapes of Wrath, Cannery Row, etc. The political interjections feel out-of-place and, often, out-of-touch, as does a shockingly racist passage near the very beginning. The Biblical allegory is about as heavy-handed as one could expect from the title. One major character is so cartoonishly evil that it’s difficult to feel invested in her storyline. Structurally, it’s all over the place, with some particularly meandering plot elements, and a narrative voice that switches between third-person and first-person when you least expect it. And yet, somehow it just works. I didn’t know quite how I felt about East of Eden until the very end, which was one of my most emotional reading experiences in recent memory. [File under Fiction / Literary.]

★★★★

A Swim in a Pond in the Rain: In Which Four Russians Give a Master Class on Writing, Reading, and Life – George Saunders
The stories are, of course, a pleasure to read, especially Tolstoy’s “Master and Man”, Gogol’s “The Nose”, Turgenev’s “The Singers” … But what I didn’t expect was just how illuminating Saunders’s commentary is to each story, and how much I learned about what goes into crafting a good short story. I’m not a writer, but a lot of the exercises and approaches here apply just as well to critically reading and enjoying fiction, and I came out of this book with a deeper appreciation of Russian literature and the short story in general. [File under Fiction / Literary / Short stories but also Literary Criticism.]

Flutes of Fire: Essays on California Indian Languages – Leanne Hinton
(Something of a re-read – I’ve read part of it before for a class in undergrad, but never read it cover-to-cover before.) This book feels like something of a rarity – a broad-strokes look at indigenous languages of California that’s accessible to the layperson while still being linguistically rigorous and not “dumbing down” the subject matter. And it’s simply a delight to read, as you can feel Hinton’s sense of wonder about the languages and language features that she’s writing about. This is the book that originally kindled my interest in linguistics when I read part of it in an introductory college course, and reading it now brings that feeling back. The last two sections, “Language and Dominion” and “Keeping the Languages Alive”, are ones that I don’t think I’ve read before and they feel particularly important now, though it’s sad to think how much has likely been lost since this book was first published in 1994. [File under Linguistics / Areal / North America.]

Linguistic Diversity in Space and Time – Johanna Nichols
There’s a lot going on here, even for a reader like me with only an undergraduate grasp of linguistics. Nichols (1) assesses a set of especially “stable” language features for nearly 200 languages covering most of the world’s language families and isolates, building probably the largest database of language typology of its time; (2) presents a novel way to trace the stability of each of these features over space and time; and (3) from there, develops a theory of language diversity that both suggests answers to long-standing puzzles (e.g. Q: Why are there so many more distinct language families in the Americas than in the “Old World”? A: The typological properties common in American languages are exactly the ones that cause the linguistic data necessary for the comparative method to “erode” away more quickly.) and raises new ones (e.g. Nichols’s data suggests that the Americas were initially populated largely by people coming from the Australian/Papuan direction, which certainly isn’t a mainstream theory). This is a truly monumental project, that easily could have ended up as multiple important books, and it is wild to see so much ambition and insight packed into a single, fairly slim, volume. [File under Linguistics / Typology.]

Bullshit Jobs: A Theory – David Graeber
At one point, Graeber describes the job of a cultural anthropologist to be (paraphrasing) digging deeply into cultural assumptions that are taken at face value, and Bullshit Jobs does just that. It examines a concept that doesn’t even feel worthy of writing a book about (“many jobs feel boring and meaningless” is hardly news!), and uses it as a lens to examine the bureaucratization of the economy, the history of popular perception of capitalism, and some surprising holdovers from the age of feudalism that seem to reassert themselves in today’s “bullshit” jobs. And unlike many such books that present series social issues without presenting much in the way of a remedy, Graeber ends with a decent argument in favor of universal basic income as at least a partial solution here. [File under Anthropology / Cultural.]

Dune – Frank Herbert
(Technically a re-read, but I haven’t read it since I was in high school, so I’ll count it.) Re-reading Dune (for the obvious reasons) was an interesting experience because, while I remembered the overall plot outline, one thing that slipped my memory was just how weird of a book Dune is, and I mean that in the best possible way. Even compared to its New Wave science fiction contemporaries, Dune is really its own thing, spending seemingly more time speculating about desert ecology, religion, and the spread of ideas than on, you know, the action itself, much of which ends up taking place “off-screen”. The plot itself feels less engaging in the second half of the book, but Herbert can be forgiven for this because everything else is still so interesting. [File under Fiction / Speculative.]

★★★½

Manhattan Beach – Jennifer Egan
A beautifully-written novel with a lot of passages that I really liked, that unfortunately seemed to sputter out partway through, with too much time spent following a gangster subplot that just felt like one cliché after another. Egan spent over a decade putting Manhattan Beach together, and her meticulous attention to period detail is clear, but in the process the novel ends up not having the same kind of wild spark that A Visit From the Goon Squad had. [File under Fiction / Historical.]

We Should All Be Feminists – Chimamanda Ngozi Adichie
Adichie makes her point clearly and elegantly. It’s a very short read, but perhaps is just the length it needs to be. [File under Essays.]

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy – Cathy O’Neil
A well-written and timely analysis of the dangers of the over-reliance on ML to do things like score résumés and process loan applications. I feel like there’s not too much here that people in the ML field haven’t been warning about for years, but it’s important that this issue gets broader attention, and this book does a great job of clearly explaining the issue to a wide audience. [File under Technology / Social Aspects]

The Chess Garden: Or, The Twilight Letters of Gustav Uyterhoeven – Brooks Hansen
This might not be my favorite novel I read this year, but it is certainly one that will stick in my memory for a long time. I don’t even really know how to describe it – a colorful biography of the (fictional) Dr. Uyterhoeven, interspersed with his stories of a journey through a land populated by anthropomorphic board game pieces, which in turn form allegories about … the conflict between rationalism and “vitalism”? This is a profoundly ambitious novel that comes very close to something special. It does end up feeling tedious near the middle, and the ending was a little too overtly “spiritual” for my taste. But nonetheless, I have no regrets about opening up this book. [File under Fiction / Experimental.]

★★½

Bay Area Cocktails: A History of Culture, Community and Craft – Shanna Farrell
Farrell has certainly done her research, conducting dozens of interviews over several years to put together this book. And it’s full of interesting tidbits about the various characters behind the Bay Area’s cocktail renaissance. What feels missing is a discussion of just what makes the Bay Area special (if anything!) - most of the trends that the author traces in cocktail history have little to do with this particular region, and as a result the book feels aimless at times. [File under Food & Drink / Cocktails / History.]

Reality and Other Stories – John Lancaster
“Old-fashioned ghost stories about technology” is a premise that I very much wanted to like, but unfortunately Lancaster fails to deliver with most of these. Most of them feel a little too clever, as though they’re written by someone who wants to write “genre fiction” but is feeling just a bit too smug about it. The best story in the collection, “Reality”, is also the one that is the furthest away from that overall premise. [File under Fiction / Speculative / Short stories.]

★½

Gem of the Lost Coast: A Narrative History of Shelter Cove – Mario Machi
A passably-written collection of local history of the Shelter Cove area, written by an active participant in that history. [File under History / California / Local.]

Native California Hero’s [sic] of the Miwok Confederation: Teleguac, Estanislas and Yolosko – Guy (Redcorn) Nixon
I was hoping to read some interesting and previously undocumented bits of Miwok history taken from oral interviews conducted by the author. Unfortunately, the end result is so poorly written as to be largely unreadable. The opening chapters, on the ecological aspects of Native Californians’ first interactions with settlers, raises some interesting questions but are difficult to take at face value. [File under History / California / Native American.]

Reading List - 2020

2021-01-09T00:00:00+00:00

[Originally posted on Facebook]

Every January, I post short reviews of the books I read the previous year. Due to the craziness of the past year, I didn’t have the energy to actually write reviews, but here’s what I read in 2020:

★★★★★

One Hundred Years of Solitude – Gabriel García Márquez (tr. Gregory Rabassa)

★★★★½

Utopia Drive: A Road Trip Through America’s Most Radical Idea – Eric Reese
The Last Samurai – Helen DeWitt
Exhalation: Stories – Ted Chiang
Piranesi – Suzanna Clarke

★★★★

The Shock Doctrine – Naomi Klein [thanks Mel]
So You Want to Talk About Race – Ijeoma Oluo
The Color of Law: A Forgotten History of How Our Government Segregated America – Richard Rothstein
Stories of Your Life and Others – Ted Chiang
Lame Fate / Ugly Swans – Arkadi and Boris Strugatsky (tr. Maya Vinokour)
Monday Starts on Saturday – Arkadi and Boris Strugatsky (new tr. by Andrew Bromfield)
October: The Story of the Russian Revolution – China Mieville

★★★½

Trysting – Emmanuelle Pagano (tr. Jennifer Higgins and Sophie Lewis)

★★★

The Tall Building Artistically Reconsidered – Ada Louise Huxtable
Lingo: Around Europe in Sixty Languages – Gaston Dorren (tr. Alison Edwards)

★★½

The Mathematician’s Shiva – Stuart Rojstaczer

★★

Give them an Argument: Logic for the Left – Ben Burgis

Migrating a ASP.NET Application from SQL Server to MySQL

2020-04-18T00:00:00+00:00

Last weekend I had the unenviable task of moving a legacy ASP.NET application to a new server, and as part of the move I had to migrate the application’s database from SQL Server to MySQL, the only SQL flavor supported by the new server.

What I thought would be a fairly routine switchover turned out to be a rather involved couple of days, and none of the resources I could find online were able to illuminate the whole process for me. So I’m writing the post I wish I could have read before embarking on this adventure.

Here’s what I did:

Step 1. Copy over the Database

To copy over the data from the old SQL Server database to the new MySQL database, I used MySQL Workbench’s Migration Wizard, connecting to the old database through a SQL Server ODBC driver.

One would hope that this would be an automatic process, but unfortunately, it didn’t quite work without manual intervention, for a few reasons:

The SQL generated by the Migration Wizard to recreate the tables failed with a cryptic “Error Code 1005. Can’t create table”, giving “Specified key was too long; max key length is 767 bytes” as he reason. It ended up being an encoding issue – for whatever reason, the generated SQL assigned an unusual character encoding to each field. This issue went away after I removed all of the ENCODING clauses in the generated SQL and re-ran it.
Timestamp literals (e.g. in default values for TIMESTAMP fields) were not correct in the generated SQL, I assume because of differences in how dates and times are represented in SQL Server vs MySQL. I had to fix these by hand.
After all this, some indices and constraints still got messed up, and needed to be manually fixed before copying over the data from the original database.

Step 2. Install New Packages

NuGet wasn’t really a thing when I first created this application (in 2011), so I wasn’t looking forward to this step, but I was pleasantly surprised by how easy NuGet is to use and how solid the Visual Studio integration for it is.

One thing that was a little tricky was that my application uses .NET Framework 4.0 (and I’ve been unable to upgrade it to 4.5+ for a variety of reasons), so I couldn’t use any of the last fewpain versions of MySql.Data (both v8 and v6.10 require .NET Framework 4.5). It took some trial and error, but I ended up installing the following packages:

MySql.Data 6.9.12
MySql.Data.Entity 6.9.12
MySql.Web 6.9.12 (for membership, see Step 4 below)
WebMatrix.WebData (for membership, see Step 4 below)

Step 3. Configure MySQL in web.config

Now that MySql.Data.MySqlClient is installed, switching over to it is actually quite painless, and requires no code changes. I made the following changes to my Web.config file:

In configuration/connectionStrings, update all the connection strings to point to the MySQL database and swap out providerName="System.Data.SqlClient" for providerName="MySql.Data.MySqlClient".
In system.data/DbProviderFactories, add the following (this may be automatically added for you by NuGet):

<add name="MySQL Data Provider" invariant="MySql.Data.MySqlClient" description=".Net Framework Data Provider for MySQL" type="MySql.Data.MySqlClient.MySqlClientFactory, MySql.Data, Version=6.9.12.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" />

Replace the entityFramework section with the following:

<entityFramework codeConfigurationType="MySql.Data.Entity.MySqlEFConfiguration, MySql.Data.Entity.EF6">
  <defaultConnectionFactory type="MySql.Data.Entity.MySqlConnectionFactory, MySql.Data.Entity.EF6" />
  <providers>
    <provider invariantName="MySql.Data.MySqlClient" type="MySql.Data.MySqlClient.MySqlProviderServices, MySql.Data.Entity.EF6, Version=6.9.12.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d"></provider>
  </providers>
</entityFramework>

In runtime/assemblyBinding, add the following:

<dependentAssembly>
  <assemblyIdentity name="MySql.Data" publicKeyToken="c5687fc88969c44d" />
  <bindingRedirect oldVersion="1.0.0.0-6.9.12.0" newVersion="6.9.12.0" />
</dependentAssembly>

And now …

After I performed steps 1–3, I tried running my application. Everything worked! I tried changing up some values in the MySQL database to make sure that we weren’t still pointing to the old SQL Server database. It picked up the new values! Well, that wasn’t so bad.

So I shut off SQL Server, reload the page, and …

A network-related or instance-specific error occurred while establishing a connection to SQL Server.

Wait, what? We’ve switched over the connection strings to use MySqlClient – how could we still be trying to access SQL Server after?

After some disbelief, I finally realized that:

While testing my changes, I (foolishly) set up the new MySQL database on the same box, with the same database name and with the same admin user credentials as the old SQL Server database (albeit on a different port) – in other words, with an identical connection string, and
I’d neglected to make any changes to the <membership> section of Web.config, so the default System.Web.Security.SqlMembershipProvider provider was still being used – and SqlMembershipProvider only supports SQL Server.

So, by a unfortunate confluence of events, even though the application was correctly connecting to the new MySQL database via MySql.Data.MySqlClient, the membership provider was reading the same connection strings as SQL Server connections and connecting to the SQL Server database that happened to be on the same box, with the same admin user credentials – a SQL Server database that had still existed in my initial tests, before I shut it down!

If, like me, you’re using SqlMembershipProvider, there’s one more step you’ll need to take:

Step 4. Switch over to MySqlMembershipProvider (if needed)

First, add the MySql.Web and WebMatrix.WebData packages with NuGet if you haven’t already.

Then, switch to MySqlMembershipProvider in the <membership> section of your Web.config:

<membership defaultProvider="MySQLMembershipProvider">
  <membership>
    <providers>
      <add name="MySQLMembershipProvider" type="MySql.Web.Security.MySQLMembershipProvider, MySql.Web, Version=6.9.12.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" connectionStringName="..." ... />
  </membership>
</membership>

You may also need to configure MySQLProfileProvider and MySQLRoleProvider in Web.config, or this may automatically be done for you by the NuGet installation. In any case, I didn’t have to do anything with these.

After doing this, my application was finally able to start up without crashing (because it no longer was trying to connect to a SQL Server database that no longer existed), but none of the user-related logic was working, as Membership.GetAllUsers() would always return an empty collection.

It turns out that MySQLMembershipProvider, sensibly, uses different tables in your database than the default SqlMembershipProvider does: e.g. instead of an aspnet_users table it uses a my_aspnet_users, and similarly all other tables are prefixed with my_aspnet_. Migrating the data over from the old aspnet_ tables was not completely straightforward either, because the schema is slightly different, with columns renamed and reordered, and in some cases a few columns added. I did some janky SQL-fu to get the user data moved over because I couldn’t find a more automated solution (old StackOverflow threads mention a ASP.Net configuration tool that could help with this user data migration, but it doesn’t seem to exist anymore).

And you’re done!

It was a slightly more perilous journey than anticipated, but after following these steps I was able to get my legacy ASP.NET v4.0 application running with MySQL. And now that it’s on a database platform with as rich an ecosystem as MySQL, hopefully I won’t have to do anything like this again.

Reading List - 2019

2020-01-10T00:00:00+00:00

[Originally posted on Facebook]

Another year, another reading list! Here’s what I read last year.

★★★★★

Evicted: Poverty and Profit in the American City – Matthew Desmond

Even if this were a work of fiction, it would get high marks for its gritty, gripping portrayal of impoverished families struggling to survive in inner-city Milwaukee. The fact that this is all documentary nonfiction makes it all the more remarkable. Desmond doesn’t offer much in the way of solutions, but he viscerally depicts the problem of extreme urban poverty. If every policymaker had to read this book, maybe things could get better. [File under Sociology / Urban.]

★★★★½

The Death and Life of Great American Cities – Jane Jacobs

Jacobs breaks down urban dynamics into axiomatic pieces and applies them masterfully in arguing for what makes a city work (short blocks, mixed uses, mixed building ages, and density) and what doesn’t (modernist planning, essentially). Her analysis has some flaws (for one thing, there’s no mention of gentrification) but has nonetheless aged remarkably well given its fast-moving topic. I can see why this book was such a big deal when it came out, and I can’t believe it took me so long to finally read it. [File under Sociology / Urban.]

The Goldfinch – Donna Tartt

One of those books you just can’t put down – I think I read the last two parts in one fell swoop. A lot of critics seem to have accused The Goldfinch of being melodramatic, and maybe it is, but what is life without some melodrama? (Admittedly, I came into it already a Donna Tartt fan – The Secret History is a favorite.) There’s some wonderful portrayals of grief and obsession, but most of all, I just appreciate a story about someone progressively digging themselves into an unthinkably deep hole, and still, against all odds, climbing most of the way out. [File under Fiction / Literary.]

★★★★

The Dwarf – Pär Lagerkvist (tr. Alexandra Dick)

Warring states, brutal betrayals, and a cunning, misanthropic dwarf. Nope, it’s not Game of Thrones, but rather Lagerkvist’s bleak novella exploring the nature of evil. [File under Fiction / Literary.]

Samarkand – Amin Maalouf (tr. Russell Harris)

Witty and poetic, as a novel about Omar Khayyam ought to be. Maalouf ties two parallel historical threads (one in the 11th century and one in the early 20th century) together in a surprisingly natural way, skillfully juggles fact and fiction, and injects contemporary relevance through an examination of European influence in 20th century Persia. [File under Fiction / Historical.]

The Art of Piano Playing – Heinrich Neuhaus (tr. K. A. Leibovitch)

Neuhaus’s book, ostensibly about piano performance, is really equal parts playing tips, musings about aesthetics, and anecdotes from his time as a fixture of Moscow’s music scene. Don’t expect a straightforward account, but there are some real gems buried under the constant (but entertaining) digressions. [File under Music / Performance.]

House of Suns – Alistair Reynolds

Reynolds manages to write a story that is both thrilling and moving, despite an unimaginably far-future, high-technology setting. This is “hard” science fiction at its best. [File under Fiction / Speculative.]

CivilWarLand in Bad Decline – George Saunders

Even in his first short-story collection, Saunders shows his unique, darkly satirical voice, but these stories in particular do come off feeling somewhat repetitive, and setting them all in bizarre theme parks doesn’t help. Highlights: “CivilWarLand in Bad Decline”, “Offloading for Mrs. Schwartz”, and the remarkable Afterword in the new edition. [File under Fiction / Satire / Short stories.]

The Conquest of Bread – Piotr Kropotkin (tr. Piotr Kropotkin)

Kropotkin turns out to be much more straightforward and unpretentious than I’d expected. The Conquest of Bread convincingly lays out an alternative vision for society, while still maintaining nuance and not getting bogged down in ideology (I particularly appreciated the chapter on the necessity of luxuries, such as pianos, even in an anarchist society). [File under Philosophy / Political.]

The Classical Style – Charles Rosen

The Classical Style takes something I thought I understood (sonata form) and shows it to be both far more complicated and more intuitive than I’d thought. This book was slow going (it took me a few months on-and-off to get through it due to all the musical examples) but completely changed how I listen to classical music, particularly that of Haydn and Mozart. Worth it for anyone looking to learn more about the nuts and bolts of classical music. [File under Music / Analysis.]

★★★½

The Murray Bookchin Reader – Murray Bookchin (ed. Janet Biehl)

Bookchin had good ideas on communalism, social hierarchy and ecology, but his rhetorical style is dense and not always comprehensible. There were a lot of thought-provoking bits, but they were too often buried by rather abstruse writing filled with not-well-defined terminology. [File under Philosophy / Political.]

The Cloven Viscount – Italo Calvino (tr. Archibald Colquhoun)

Not Calvino at his best, but short and sweet, reading almost like an offbeat fairy tale. [File under Fiction / Fantasy.]

Energy and Civilization – Vaclav Smil

The first half of the book (prehistoric agriculture until the fossil fuels era) is fantastic and informed a lot about how I think about human prehistory. Everything after that is a tedious slog as Smil seemingly tries to cram in every single invention and development of the Industrial Revolution into his book for the sake of comprehensiveness, without much payoff at the end. I would stop after Chapter 4. [File under History / Energy.]

A Gentleman in Moscow – Amor Towles

A clever premise, hampered by a charming but ultimately dull protagonist and a generally one-dimensional cast (with some notable exceptions). The novel is jam-packed with references to Russian literature, but I didn’t get the sense that Towles actually succeeded in understanding the “Russian mindset”, so to speak. If you’re in the mood to read a contemporary multi-generational epic set largely in the Soviet Union, try Sana Krasikov’s fantastic debut novel The Patriots instead. [File under Fiction / Historical.]

The Mysteries of Pittsburgh – Michael Chabon

A modern coming-of-age story that’s by turns poignant and hilarious. It definitely has its moments, but the plot feels uneven, with the fantastical gangster subplot seeming particularly out of place. I prefer Chabon’s later works, but it’s interesting to see how he’s grown as a writer. [File under Fiction / Contemporary.]

★★★

Fear and Loathing in the North: Jews and Muslims in Medieval Scandinavia and the Baltic Region – ed. Cordelia Heß, Jonathan Adams

I picked this up on a whim (the Kindle edition was free) and boy, did I end up learning far more than I ever expected about Jews and Muslims in medieval Scandinavia and the Baltic region. There’s some interesting scholarly work here, and it stoked my interest in the Medieval period, leading me to pick up The Cloven Viscount (Calvino), The Dwarf (Lagerkvist), and Samarkand (Malouf). [File under History / Northern Europe.]

My Boyfriend is a Bear – Pamela Ribon and Cat Farris

A cute, quick read. [File under Graphic novels / Humor.]

Killing Commendatore – Haruki Murakami (tr. Philip Gabriel and Ted Goossen)

I wanted to like this book, and the painting scenes are some of the best depictions of the artistic process I’ve read in fiction. In the end, though, it felt like a slog, and had none of the payoff that I was hoping for – nothing seemed to get resolved and not much was explained. Even more so than 1Q84, this felt like a knockoff Murakami novel, rather than the real deal. [File under Fiction / Literary.]

Childhood's End – Arthur C. Clarke

An interesting subversion of the alien-invasion trope, but the plot is all over the place, jumping from sci-fi to crime drama to utopian exposition to a bizarre ESP subplot that Clarke himself later regretted including. The Strugatsky brothers delivered on this premise better in their works, especially their 1987 novel The Ugly Swans. [File under Fiction / Speculative.]

Visualizing Classical Music Influence with networkx

2019-02-27T00:00:00+00:00

See https://nbviewer.jupyter.org/github/AlexNisnevich/blog/blob/master/_notebooks/music-graphs.ipynb

Reading List - 2018

2019-01-06T00:00:00+00:00

[Originally posted on Facebook]

Here’s what I read last year:

★★★★½

Dictionary of the Khazars: A Lexicon Novel (“female” edition) – Milorad Pavič (tr. Christina Pribicevic-Zoric)

This is exactly the kind of fun, eccentric modernist fiction that I love so much: a novel in dictionary form, with the entries telling a richly interlinked narrative spanning three time periods, loosely inspired by the 9th century mass conversion of the Khazar people. Magical realism with an emphasis on the “magical”, it’s all written in a dreamy, exuberantly fantastical voice, as though it were a fairy tale recounted by a really good storyteller.

The Patriots – Sana Krasikov

A contemporary equivalent to those great Russian epics of the nineteenth century. The Patriots is a sprawling novel, zigzagging back and forth between Russia and America and between past and present, as it tells the tale of one family’s impossibly complex history. The narrative has its share of melodrama, but it doesn’t lose its sense of universality, and I felt that I came out of it with a newfound understanding of (and appreciation for) my own family.

★★★★

Amusing Ourselves to Death: Public Discourse in the Age of Show Business – Neil Postman

Postman’s thesis that our ever-more-sophisticated entertainment industry is drugging society into helplessness doesn’t feel particularly novel, but he argues his point masterfully. Particularly intriguing was his exploration of the importance of rational argument in the Age of Reason (e.g. his comparison of the Lincoln-Douglas debates to political debate today). Postman doesn’t offer much in the way of concrete solutions, but this book is more timely than ever (and his analysis of Reagan as the “TV entertainer president” is eerily prescient…). [thanks Asali]

Earthsea (first trilogy) – Ursula K. Le Guin

(A Wizard of Earthsea ★★★★, The Tombs of Atuan ★★★½, The Furthest Shore ★★★½)

I can’t believe I waited so long to start reading the Earthsea books! This is Fantasy at its best: a depiction of a truly unique world (not just Middle-Ages-plus-pointy-ears), used as a backdrop for an exploration of serious themes through a humanist lens, rather than a sequence of mindless violence.

Pale Fire – Vladimir Nabokov

A bold experiment in form, Pale Fire is a novel presented as a collection of lengthy, digressive annotations to a (fictional) epic poem. It’s also perhaps one of Nabokov’s most autobiographical works, in its own way – it’s hard not to see Nabokov himself in Kinbote, his refugee literature professor with a mysterious past. Though it’s odd, then, that Nabokov makes his protagonist so unsympathetic – ultimately, a deranged stalker. Still, the rich, opinionated narrative voice makes it a joy to read.

The Great Pianists – Harold C. Schonberg

Schonberg attempts a doubly-impossible feat – to (1) compare how the great pianoforte players, from Mozart to Gould, have approached the art of piano playing, despite a lack of clear sources, and (2) to then explain his results in a readable way – and yet not only succeeds, but also in the process presents the lives of these pianists in such an engaging way that I’d say that anyone interested in music would appreciate this book.

White Noise – Don DeLillo

DeLillo satirizes academia by making every character in White Noise, young and old, go through the novel as though they are an indifferent sociologist observing the world around them. The characters – a professor of “Hitler studies” and his family and colleagues – aren’t very relatable figures as a result, but it does make for a hilarious, incisive work, at least until it starts to lag due to the introduction of a more “conventional” (relatively speaking) plot line in the third act. , young and old, go through the novel as though they are an indifferent sociologist observing the world around them. The characters – a professor of “Hitler studies” and his family and colleagues – aren’t very relatable figures as a result, but it does make for a hilarious, incisive work, at least until it starts to lag due to the introduction of a more “conventional” (relatively speaking) plot line in the third act. [thanks Danielle]

All the Names – Jose Saramago (tr. Margaret Jull Costa)

Saramago does what he does best: present the smallest, most insignificant details in a character’s life in such a way that they seem as weighty and important as the actions of a classical hero. At the same time, he describes bureaucratic procedure so eloquently that the bureaucracy in question (a Central Registrar of records for an unnamed city) becomes a central character in its own right. A beautiful little novel hampered only by a clumsy, abrupt ending.

Some Trick: Thirteen Stories – Helen DeWitt

No doubt informed by DeWitt’s personal struggles, the stories contained in Some Trick are full of scathing tear-downs of artistic establishments of all sorts, be they publishers, galleries, tabloid journalists, or pompous academics. But the overarching theme is still a hopeful one: despite all the struggles and sacrifices, art will still go on, if only because artists simply can’t stop doing art. Highlights: “On the Town”, “The French Style of Mlle Matsumoto”.

The Snail on the Slope – Arkady & Boris Strugatsky (tr. Olena Bormashenko)

Not my first time trying to read The Snail on the Slope, but the first time that I felt like I “got” it, no doubt thanks to Bormashenko’s excellent new translation. Boris later described this novel as a meditation on the conflict between humanity’s past, present, and future. It may be that, but it is also an excellent Kafkaesque tale alternating between two protagonists: one desperately trying to enter an enigmatic forest despite the objections of a faceless bureaucracy, and the other trying just as desperately to escape the forest.

Gravity's Rainbow – Thomas Pynchon

I’m torn. i want to hate Gravity’s Rainbow because of the months of arduous reading that it took to get to the end of it – but how could I hate this book? It’s by turns hilarious and tragic, and so dense with allusions to everything under the sun that I feel like I understood half of it, at best, even so, the half that I did understand made it worthwhile. Despite its absurdity, it’s perhaps one of the most realistic fictional depictions of the military-industrial complex and the horrors of war. But I’m not sure. I’m still not done processing what I’ve read. [thanks dad]

★★★½

Seven Days in the Art World – Sarah Thornton

If you still have any romantic preconceptions about the role of an artist today, this is the book to dispel them. Thornton masterfully weaves together interviews and observations of all the different interlocking circles underpinning the art world, from collectors to critics, to, finally, and almost as an afterthought, the artists themselves. I certainly can’t look at contemporary art the same way anymore. [thanks Emily]

Leviathan Wakes – James S. A. Corey

Fun, fast-paced space noir in a richly crafted near-future universe with a cynical vibe. I couldn’t stop turning pages while reading it, but nonetheless I don’t feel a strong urge to read the sequels. [thanks Jacob]

In Search of the Hunters and their Tribes: Studies in the History and Culture of the Taiwanese Indigenous People – ed. David Faure

There isn’t much English-language books out there on Taiwanese indigenous people, but fortunately this is a good one: a fascinating, meticulously-researched collection of essays on the topic of identity among indigenous tribes in the past and present. I was especially struck by Kai Yiu Chan’s essay tracing the history of plastic bead manufacturing and its effects on the social structure of the Paiwan and Rukai tribes.

The Displaced: Refugee Writers on Refugee Lives – ed. Viet Thanh Nguyen

A timely collection of powerful accounts of refugee experiences around the world, told from a wide variety of different perspectives, with tone ranging from defiant to heartbreaking. Highlights: “Guests of the Holy Roman Empress Maria Theresa” by Lev Golinkin, “God’s Fate” by Aleksandar Hemon, “The Ungrateful Refugee” by Dina Nayeri. [thanks Asali]

★★★

The Piano Trio: Its History, Technique, and Repertoire – Basil Smallman

A comprehensive, if dry, account of the history and theory of my favorite chamber music form.

Blocked: Stories from the World of Online Dating – ed. Ally Schwed

At their best, these short comics provide a unique, intimate look at the realities of modern dating. But too many of them just devolve to cliché.

★★½

The Denial of Death – Ernest Becker

I had high hopes for Becker’s magnum opus tracing most human behavior to our innate fear of death. And it did start off strong with its exploration of the “immortality project” concept in the first few chapters (albeit working with an outdated view of mental illness). Then, for seemingly no reason, came some bizarre chapters attempting to psychoanalyze Freud himself, and the whole thing ended on a religious note that seemed rather a cop-out.

The Soloist – Mark Salzman

A pretty good character study of an aging musical ex-prodigy, inexplicably crossed with a ridiculous courtroom drama fearing embarrassingly dated portrayals of Zen Buddhism and a cringeworthy romance subplot.

Very Special Relativity: An Illustrated Guide – Sander Bais

An interesting attempt to explain special relativity entirely using geometric reasoning with spacetime diagrams, but it didn’t really work for me. Around halfway through it all started to fall apart and made me wish that I could see some equations for a change, rather than the fiendishly complicated diagrams that ensued. [thanks Greg]

★

Roadside Picnic Revisited – Michael Andre-Druissi

The author spends six essays saying absolutely nothing of substance. Good English-language analysis of the Strugatsky brothers’ literature does exist (see, e.g. Istvan Csicsery-Ronay’s essays), but this ain’t it.

Creating Land-Use Maps with an HP 7475A Plotter

2018-09-15T00:00:00+00:00

(Or, the week I finally went off the hipster deep end)

About six months ago, I read Tobias Toft’s excellent article “An intro to Pen Plotters”. Inspired by the wonderful art he demonstrated, I set out to get my own plotter - a drawing robot from the pre-printer age - to play with.

This is the story of my first experiment with pen plotters – plotting a land-use map of Downtown Berkeley:

Setting up and using an HP 7475A plotter

I took Tobias’s advice and took to eBay to look for HP 7xxx-series plotters. I found a cheap HP 7475A that looked to be in good condition, and eagerly awaited its arrival … only to discover that I hadn’t been careful enough in looking at the listing and that the plotter came with the dreaded HP-IB port:

I didn’t know this at the time, but it turns out the that HP 7475A was one of the few HP pen plotters that came in two models: one with a forward-compatible serial port and one with a backward-compatible HP-IB port. Not having any idea what to do with this strange plug (it’s not 1985 anymore, after all), I turned to the internet and discovered a post that began a sentence with “FWIW, if you have an HP-IB plotter, you should probably just hack an Arduino into the motor control connections…” Yikes, that’s a bit more than I’d signed up for.

Fortunately, the eBay seller was understanding and let me return the HP-IB plotter. Eventually I managed to find an HP 7475A with a serial port. From there, I was able to follow Tobias’s instructions. I got a null modem cable and a serial-to-USB converter cable and plugged the plotter into my Macbook.

I tested the plotter out by sending some commands over Coolterm, but found it pretty finicky. After trying a few different terminal emulators and plotting utilities, I discovered the excellent Chiplotle library, and have been using it ever since. For interactive plotting I load the chiplotle shell, and for quickly plotting HPGL files created separately I use the plot_hpgl_file.py script.

Loading and Exporting Shapefiles

The first step to plotting a map is loading the appropriate shapefile in a GIS environment (I use the open-source QGIS) and exporting each desired layer as PDF. I followed the instructions in the “Preparing Data” section of the QGIS tutorial to load a map of Berkeley from OpenStreetMap:

From there, I exported a PDF for each layer I was interested in:

roads.pdf is the set of lines (roads, train tracks, paths, etc) with labels removed
comm.pdf is the union of the “commercial” and “retail” polygon layers
inst.pdf is the “institutional” polygon layer (admittedly this layer doesn’t have a lot in it)
green.pdf is the union of the “farmland”, “forest”, “grass”, and “recreational_ground” polygon layers

Initially I separately exported the “residential” layer as well, but I found that it cluttered up the map too much. There is also a layer consisting of all structures, which likewise would have been cool to use (in particular, it would have made the university grounds on the right side look less sparse) but likewise it caused too much clutter.

I selected each layer (or set of layers) in turn and exported to PDF (Project | Export to PDF), being careful to preserve the exact same dimensions and map coordinates between each PDF.

Converting PDF -> HPGL

Now that I had a bunch of PDFs, I needed to convert them to the HPGL format for plotting. I couldn’t find any way to convert PDF -> HPGL in one operation, and settled for converting PDF -> PS using pdf2ps and PS -> HGPL using pstoedit.

I wrote a simple pdf2hpgl.sh script that, given a list of pairs of PDF file paths and pen numbers (for example, pdf2hpgl.sh roads.pdf 3 comm.pdf 5 inst.pdf 4 green.ps 2), converts each PDF file into an HPGL file, changes all SP (select pen) commands to use the given pen number (instead of the more-or-less random pen selections made by pstoedit), and concatenates the result together into one big HPGL file:

#!/usr/bin/env bash
# Requires pdf2ps and pstoedit.
OUTPUT_FILE="out.hpgl"

rm $OUTPUT_FILE

while (( "$#" >= 2 )); do
  PNG_FILE=$1
  PEN=$2
  shift 2

  PS_FILE="${PNG_FILE%.*}.ps"
  HPGL_FILE="${PNG_FILE%.*}.hpgl"

  pdf2ps $PNG_FILE $PS_FILE
  pstoedit -f plot-hpgl $PS_FILE $HPGL_FILE
  cat $HPGL_FILE \| sed -e "s/SP[0-9]*;/SP$PEN;/g" >> $OUTPUT_FILE

  rm $PS_FILE
  rm $HPGL_FILE
done

And voila, we now have an HPGL file that we can send directly to the plotter.

Here’s what it looks like with just the roads layer plotted:

Unfortunately, the polygon layers don’t end up looking very good when simply overlaid on top of the roads layer (I neglected to take a photo of this; you’ll have to take my word for it), since they’re just rectangular outlines that are difficult to see next to the black lines of the roads.

If we want to display the land-use layers in a legible way, we’ll need to get a little creative. How about turning those polygons into hatching lines?

Creating Hatching with Adobe Illustrator

I decided to use Adobe Illustrator to generate the hatching lines, but me not being an Illustrator expert by any means, I spent many hours fruitlessly trying different tools to no avail. The issue was that I needed to create a PS file with just the lines themselves – no hatching patterns defined, no masking layers – because the HPGL dialect spoken by the HP 7475A (as opposed to the later, more sophisticated HP-GL/2) doesn’t support any of these fancy features.

I finally gave up on figuring this out on my own, and asked Stack Exchange, where I got my answer.

The workflow I finally settled on is a bit convoluted, but it does work:

Load the PS file corresponding to a single layer in Illustrator (using pdf2ps to generate the PS from the QGIS PDF export).
Remove the background layer to leave just the layer with the polygon paths.
Select the remaining layer. If there are multiple paths, use Pathfinder | Unite.
With the layer still selected, create a compound path with Object | Compound Path | Make.
Fill the path with a pattern I made of diagonal strokes, and remove the stroke outline.
Adjust the spacing of the hatching lines as desired with Object | Transform | Scale (with only “Transform Patterns” checked).
Convert the compount path to a group of hatching lines with Object | Expand and then Object | Path | Outline Stroke.
Release the clipping mask: Object | Clipping Mask | Release.
Select everything and click Pathfinder | Crop in the Properties panel.
Finally, export the result back to PS by printing to Postscript using the “Device Independent” PPD. I then converted the PS file to HPGL using a modified version of my pdf2hpgl.sh script that simply skips the pdf2ps step.

Here’s what it looks like when I plot the resulting hatch lines for just one layer (the “commercial” layer):

Not bad, huh?

Once we have hatching for one layer working, we can do the rest of them just by repeating the process for each layer. I did have to carefully align each layer together in Illustrator (and then print them out to Postscript one at a time by selectively hiding layers):

That said, you can create some cool glitchy art by neglecting to properly align the layers:

The Final Product

I ran plot_hpgl_file.py and crossed my fingers. After 10 minutes of meticulous plotting, the map came out of the plotter, and actually looked halfway decent. I added some text and carefully cut the paper from 11”x17” to 11”x14” to reduce the amount of unsightly white space at the bottom:

Not perfect, but it’s certainly not bad for a week’s work.

I haven’t had the chance to do much else with the pen plotter yet, but hopefully this gave you a taste of what these little robots can do. And who knows, maybe I can inspire someone else to save a plotter from a junkyard, like Tobias Toft’s article inspired me. (Just watch out for the dreaded HP-IB port…)

Gator Album Release

2018-08-19T00:00:00+00:00

After 2½ years of recording, re-recording, production, mixing, and mastering, my swamp metal band’s album is finally out!

Gator is an eclectic mix of heavy metal, psychedelic rock, and classic blues. I have a feeling you’ll like it.

The album is available on Spotify …

… as well as Bandcamp and Soundcloud.

If you like what you hear, pre-order the vinyl! We’re doing a very limited press of 200 copies.

Reading List - 2017

2018-01-11T00:00:00+00:00

[Originally posted on Facebook]

Per tradition, here are all the books I read last year.

Unfathomable City: A New Orleans Atlas – Rebecca Solnit & Rebecca Snedeker

★★★★½. While Infinite City revealed fresh secrets of a city I already knew, Unfathomable City served as an unorthodox travel guide. Reading it on the plane to New Orleans, I was struck by the deep sense of love for the city that I could feel in each of the essays and maps, and the bustling energy evoked by the book was contagious.

Ghostwritten – David Mitchell

★★★★. My first David Mitchell novel, and hopefully not my last. I loved some of the interlocking segments (particularly “Holy Mountain” and “Cape Clear Island”), didn’t care for a few of them (especially “London”), and had mixed feelings about the spectacular ending. [thanks Linchuan]

A People's History of the United States – Howard Zinn

★★★★½. It’s become almost a meme in certain circles, but this is nonetheless a remarkable book. It’s not just a different perspective on history, but a different way of looking at history altogether: as the actions of ordinary people fighting for better lives rather than the whims of a few “great men”. Tolstoy would be proud. [thanks Asali]

Speaking American: How Y’all, Youse, and You Guys Talk – Josh Katz

★★★. A neat visualization of a massive study. I just wish that there was some analysis to accompany it. [thanks mom & dad]

The ABCs of Socialism – ed. Bhaskar Sunkara

★★★★. Jacobin’s first book-length publication is a slim volume that rebuts common misconceptions about socialism in a clear, no-nonsense prose that’s anything but preachy. Its design is fantastic as well, from Wriggleworth’s cartoon illustrations to the simple question-and-answer bookends for each chapter.

Absolutely on Music – Seiji Ozawa & Haruki Murakami (tr. Jay Rubin)

★★★★ Murakami, an avid classical music fan with no musical background, is the perfect foil to Ozawa, the knowledgeable yet affable conductor. Reading their casual chats about Beethoven and Mahler is great fun.

BART: A Dramatic History – Michael C. Healy

★★★★. An engaging account of BART’s history and operations, full of amusing anecdotes and overly excited chapter titles like “1977 and 1978 See Several Improvements and Added Services”. Healy was BART’s chief spokesperson for 32 years, and, while his insider status introduces a lot of bias, it also allows him to pepper the book with behind-the-scenes details.

The Shape of Things – Neil LaBute

★★★. An interesting conceit, it was difficult to care about such incredibly superficial characters. [thanks Asali and JP]

The City Shaped: Urban Patterns & Meanings Through History – Spiro Kostof

★★★★½. Kostof argues that traditional histories of urban design make the fatal flaw of assuming a linear progression of styles, and presents an alternative history based on morphology. In five chapters (“Organic” Patterns; The Grid; City as Diagram; The Grand Manner; The Urban Skyline), Kostof describes the development of five approaches to city-building, their patterns and variations, and the meanings ascribed to them throughout history. Through it all, he writes with such an engaging voice that you feel like you’re getting a tour of the world’s cities rather than reading a history book. I can’t look at a city the same way after reading it.

Snakes and Ladders: Glimpses of Modern India – Gita Mehta

★★★½. A chaotic and exciting book about a chaotic and exciting country. In her essays, Mehta gives a whirlwind tour of India throughout the 20th century, covering politics, economics, spirituality, and culture, but the book really shines when she recounts her stories of growing up with independence-activist parents in the 1940s. [thanks Asali]

How to Be a Jewish Mother – Dan Greenberg

Cute but dated vintage humor book.

The Visual Display of Quantitative Information – Edward Tufte

★★★★. The book is visually gorgeous, and Tufte’s basic laws of information design (in particular the “data-ink ratio”) provide a solid framework for reasoning about information graphics. But I didn’t learn as much as I could have – the book was short and devoted too much time to some pet techniques that didn’t seem particularly useful nowadays.

Secondhand Time: The Last of the Soviets – Svetlana Alexievich (tr.Bela Shayevich)

★★★★★. If you’re going to read any book about contemporary Russia, make it this one. Alexievich writes what she calls “novels in voices”: stories told entirely through edited transcripts of hundreds of oral interviews. She’s written about the Afghan war and Chernobyl, but Secondhand Time is arguably her most ambitious piece: a sprawling account of the first two decades of post-Soviet Russia and the disillusionment Russians felt after the collapse of the Soviet Union. She interviews everyone from gulag prisoners to NKVD interrogators, from jet-setting business-people to Central Asian migrant workers. It’ll make you laugh, it’ll make you cry, it’ll make you feel a profound sense of empathy for a country in transition.

The New Urbanism: Toward an Architecture of Community – Peter Katz

★★★½. There are some nice essays by the New Urbanism pioneers (Calthorpe, DPZ, et al), but what really makes this book is the case studies of New Urbanism communities of all shapes throughout the U.S., from new towns like Seaside to revitalization projects like downtown Providence, all complete with beautiful city plans and architectural examples. A great source of inspiration.

Lincoln in the Bardo – George Saunders

★★★★. Imagine that a novella, a play, and a nonfiction biography had a bastard child. The action takes place in a cemetery, and, though it’s ostensibly about Abraham Lincoln, he only appears for a few pages. It’s difficult to describe what Lincoln in the Bardo is exactly, but whatever it is, it’s gripping, poignant, and endlessly inventive.

Hope in the Dark – Rebecca Solnit

★★★★. In this collection of essays, Solnit argues that hope is just as important as fear in driving activism. Though most of the book dates back to 2004, it’s as important a lesson now as ever.

The God of Small Things – Arundhati Roy

★★★★★. The most beautiful and heartbreaking novel I read this year. Roy’s descriptions of life in rural Kerala are so vivid that you can practically smell the red banana trees. The story is intricately plotted, jumping backward and forward in time as the tale of the Kochamma family unfolds, and it keeps you hoping against hope even as a mounting sense of dread keeps building into an horrific climax. It’s that rare sort of novel that not only sticks with you, but makes you feel like somehow it’s been a part of you all along. [thanks Asali]

The Works: Anatomy of a City – Kate Ascher

★★★½. An enjoyable overview of the transportation, energy, communications, and waste infrastructure of New York City filled with marvelous diagrams. No profound insights here, but it’s a pleasure to leaf through and chock-full of surprising facts. [thanks Lily]

Kasimir Malevich and Suprematism – Gilles Néret

★★★½. An engaging and meticulously researched overview of an under-appreciated modern artist. Malevich is a somewhat enigmatic figure, but Néret does an admirable job of explaining his philosophy. [thanks mom]

Where'd You Go, Bernadette – Maria Semple

★★★★. A damn good satire of the tech industry, suburban life, the art world, and everything in between. Semple hooks you in with her goofy humor and it’s not until you’re halfway in that you suddenly realize how much you’ve cared about all these flawed yet lovable characters all along. I couldn’t put it down, even through some questionable plot developments near the end. [thanks Asali]

The Yellow Shadow – V M Steele

★½. The casual racism at the core of the story doesn’t age well at all, and the whole plot is frankly ridiculous.

Nonstop Metropolis: A New York City Atlas – Rebecca Solnit & Joshua Jelly-Shapiro

★★★★. A lot of good pieces (I particularly liked the ones on the evolution of hip-hop), but overall it was more repetitive and less cohesive than its predecessors. The start was particularly shaky, with the first 5-6 maps all tackling similar ideas.

Best Russian Short Stories – Thomas Seltzer, ed.

★★★★. A comprehensive and surprisingly well-curated collection of Russian short stories from the 1920s, featuring both the heavy-hitters like Tolstoy and Gorky and relative unknowns like Saltykov-Shchedrin and Sologub, all in lively, readable translations. Highlights: “The Queen of Spades” (Pushkin), “The Overcoat” (Gogol), “The Bet” (Chekhov), “Lazarus” (Andreyev). [thanks Asali]

Baudolino – Umberto Eco (tr. William Weaver)

★★★★. A lighthearted medieval adventure tale told in the style of medieval adventure tales, and an exploration of languages (much of the novel is written in a garbled pseudo-Latin), the complicated role of religion in medieval politics, and how objects obtain meaning (much of the adventure involves religious relics, real and fictional). It’s not hard to see why I enjoyed reading it so much. [thanks, Bakery Bar in NOLA]

Pastoralia – George Saunders

★★★★. Six black comedy stories from the master. It seems that Saunders was still finding his voice in these stories: he doesn’t demonstrate quite the linguistic sureness that he has in Tenth of December and a couple of these stories didn’t speak to me at all. But it’s still Saunders, and “Pastoralia” and “Sea Oak” rank among his best. [thanks Danielle]

Capital in the Twenty-First Century – Thomas Piketty (tr. Arthur Goldhammer)

★★★★½. Finally, an approach to economics that interests me. Piketty throws out the conventional methodology of working up from abstracted models and instead employs a data-driven technique informed largely by historical records. In 12 chart-heavy chapters, he traces the development of the income-capital ratio and the distribution of income and capital throughout the world (primarily France and the U.S.), and argues that the economy trends naturally towards greater wealth concentration. These chapters diagnosing the state of capital are engaging and have given me a great deal to think about, but his final chapters suggesting potential solutions felt rhetorically weak, seeming almost like an afterthought. Piketty struck me as a far better economist than politician.

The Watcher and Other Stories – Italo Calvino (tr. Weaver & Colquhoun)

★★★. “The Argentine Ant” is a great little horror tale, but I didn’t care much for the other two stories. As far as early Calvino stories go, I much preferred the ones collected in Difficult Loves.

The Leningrad Rock Scene, in Five Albums

2017-09-14T00:00:00+00:00

[Originally posted on Facebook]

Though largely unknown in the West, Leningrad in the 1980s was a center for Soviet underground rock music.

Rock music existed in the Soviet Union ever since the first Western bootleg tapes were smuggled across the border, but two pivotal events in the early 1980s enabled the Leningrad scene to break out: the 1980 Tbilisi Rock Festival, in which the Leningrad band Aquarium’s outlandish stage antics turned them into Soviet counter-cultural celebrities overnight; and the 1981 opening of the Leningrad Rock Club, under the watchful patronage of the KGB.

Leningrad rock groups existed in a fickle and uneasy peace with the authorities. They could attract huge crowds at the Leningrad Rock Club, but their lyrics were strictly controlled. In 1984, Chernenko ordered a crackdown on rock music, accusing musicians of “moral sabotage”, while the next year, Gorbachev’s perestroika offered rock musicians unprecedented freedom. Even then, freedom came with a price, as bands coming out of the underground were accused by diehard fans of selling out.

As a result, Leningrad rock musicians had to grapple with a whole host of contradictions. How could they stay true to Russia’s musical traditions while absorbing the influence of the Western rock music so easily accessible across the Finnish border? How could they write lyrics that reflected the feelings of their contemporaries without catching the attention of the authorities? And how could they get the benefits of official music-promotion channels without earning the ire of longtime fans?

Here are five of my favorite albums (in chronological order) by Leningrad rock bands, highlighting the breadth of the scene, from folk to punk to prog. Each of these bands came up with their own solution to these contradictions, and established a unique style in the process.

45 (1982)

by Кино [Kino / Film]

Listen: [Spotify] [iTunes] [Youtube]
RYM tags: Contemporary Folk / Post-Punk / Psychedelic Folk

The Album: Kino would go on to become the iconic Soviet band of the 80s, with their frontman, Victor Tsoi, becoming a cult hero. But you could hardly tell from listening to their unassuming debut album, featuring just Tsoi and guitarist Rybin, augmented by several musicians from Aquarium and a drum machine.

The Sound: Tsoi’s goal was a “synthetic” rock sound, which Kino would end up perfecting on their later albums. But due to lack of personnel, 45 instead ended up with a heavy folk sound, with sparse guitars and vocals reminiscent of the countercultural Russian folk artists known as the bards.

The Lyrics: Earnest depictions of the lives of urban youth (“Lots of time, but no more money and no one wants to let me in”), that occasionally veers into metaphor (“I’m planting aluminum cucumbers / On a field of canvas”). [English translations here.]

Highlights: [1] “Время есть, а денег нет” (“Lots of Time, But No Money”), [2] “Просто хочешь ты знать” (“You Just Want to Know”), [6] “Бездельник 2” (“Layabout #2”), [12] “Когда-то ты был битником” (“You Were Once a Beatnik”)

See also: Aquarium’s 1982 Акустика (Acoustics), another folk-inspired rock album from the same year, featuring many of the same musicians in a vastly different context. [Spotify] [iTunes] [Youtube] [English lyrics]

Радио Африка [Radio Africa] (1983)

by Аквариум [Aquarium]

Listen: [Spotify] iTunes Youtube
RYM tags: New Wave / Psychedelic Rock / Art Rock / Experimental Rock / Sound Collage / Field Recordings

The Album: Aquarium bribed the engineers of a Soviet mobile recording studio and enlisted a who’s-who of Leningrad avant-garde musicians to record Radio Africa, their first concept album. It proved to be such a sensation that the Soviet record label, Melodiya, issued an official, albeit censored, version of it a few years later.

The Sound: Progressive rock meets new wave meets world-music-before-there-was-world-music, in tracks with colorful titles like “Captain Africa” and “Tibetan Tango”, stitched together by the sounds of radio static.

The Lyrics: All over the place, ranging from sincere ballads to complete nonsense, and frequently a mix of the two (“There are books for the eyes, and books in the form of a pistol / Sit by the window and listen to the noise of great ideas / But if you are young, then you are a fierce opponent of light / This is another plus to the songs of exhausting people”). [Some English translations here.]

Highlights: [2] “Капитан Африка” (“Captain Africa”), [3] “Песни вычерпывающих людей” (“Songs of Exhausting People”), [6] “Рок-н-ролл мертв” (“Rock’n’Roll is Dead”), [10] “Время Луны” (“Time of the Moon”)

See also: Aquarium’s follow-up album, Aquarium’s follow-up album, День серебра (Silver Day), which moves away from sound collage and goes into a more old-fashioned psychedelic rock in the vein of Sgt. Pepper. [Spotify] [iTunes] [Youtube] [English lyrics]

Переезд [Pereyezd / Move-in] (1983)

by Nautilus Pompilius

Listen: [Spotify] [iTunes] [Youtube]
RYM tags: Post-Punk / Art Punk / Hard Rock / Gothic Rock

The Album: Several years before they moved to Leningrad, Nautilus Pompilius kicked off the “Ural rock scene” in Sverdlovsk with this album, a collection of various recordings from ‘82 and ‘83.

The Sound: The first side, recorded in 1983, is mostly lush and melodic, with overlapping guitar lines and passionate falsetto vocals. The second side, recorded in 1982, is noisy and punky, at times reminiscent of the Velvet Underground.

The Lyrics: Imaginative and conceptual, in the art-rock tradition. There are songs about hawks, about flying ships, about the opera, and even a song entitled “Battle Against the Tycoon”, which is about what you’d expect. [Some English translations here.]

Highlights: [4] “После и снова” (“After and Again”), [7] “Летучий фрегат” (“Flying Frigate”), [9] “Ястребиная свадьба” (“Hawk Wedding”), [10] “Анабасис” (“Anabasis”)

See also: Nautilus Pompilius’s 1986 breakout album Разлука (Separation), which sees them confidently adopt a synthpop aesthetic and features a number of their mid-career hits, such as “Chained Together” and “Alain Delon”. [Spotify] [iTunes] [Youtube] [English lyrics]

Ночь [Noch / Night] (1986)

by Кино [Kino / Film]

Listen: [Spotify] [iTunes] [Youtube]
RYM tags: Post-Punk / Jangle Pop / New Wave

The Album: Noch was the first album of Kino’s “classic” (‘86-’90) lineup, which saw the band completely sever its ties to Aquarium and go off into its own direction. It was this album, along with the band’s appearance in the cult film Assa the following year, that placed them into the cultural mainstream.

The Sound: With Noch, Kino finally achieved the post-punk sound that Tsoi envisioned, with funky bass lines and guitar rhythms by turn menacing and danceable. Think Joy Division, but more upbeat.

The Lyrics: Relatable lyrics about contemporary life, ranging from playful (“We have cigarettes and matches, and a bottle of wine / We saw the night, we walked all night long) to cynical (“You go to the kitchen, But the water here is bitter, / You can’t sleep here, and you don’t want to live here. / Good morning to you, our last hero!”). [English translations here.]

Highlights: [2] “Фильмы” (“Movies”), [3] “Твой номер” (“Your Number”), [6] “Последний герой” (“The Last Hero”), [11] “Мы хотим танцевать” (“We Want to Dance”)

See also: Kino’s iconic 1989 album Звезда по имени Солнце (A Star Called the Sun), a gloomy post-punk masterpiece recorded by the band at the peak of their popularity and shortly before Victor Tsoi’s tragic death. [Spotify] [iTunes] [Youtube] [English lyrics]

Песня о безответной любви к Родине [Song about an Unrequited Love for the Motherland] (1991)

by Ноль [Nol’ / Zero]

Listen: [Spotify] [iTunes] [Youtube]
RYM tags: Folk Punk / Russian Folk Music

The Album: Nol’, one of the youngest bands in the scene, skyrocketed into prominence with this topical album of nostalgic folk-punk, released just as the Soviet Union was about to collapse. Their popularity was short-lived, however, as their frontman, Fyodor Chistyakov, was imprisoned the following year for the attempted murder of his girlfriend.

The Sound: Imagine the sound of classic rock’n’roll, but with the guitars largely replaced by accordion and mandolin. Basically, Leningrad’s answer to the Pogues.

The Lyrics: Themes include post-Soviet depression (“Instead of a blue sky, there’s a grey ceiling”), coping strategies (“Having smoked hashish / Life is becoming beautiful”), and the ever-present legacy of authoritarianism (“You ask me why sometimes I’m silent / Why I don’t laugh, why I don’t smile / … / It’s just that I live on Lenin Street / And I freak out from time to time.”) [Some English translations here.]

Highlights: [1] “Этот русский Rock-n-roll” (“This Russian Rock’n’Roll”), [3] “Улица Ленина” (“Lenin Street”), [5] “Иду, курю” (“I Walk and Smoke”), [9] “Человек и кошка” (“A Man and a Cat”)

See also: DDT (ДДТ)’s 1989 album Я получил эту роль (I Won This Role), a cynical hard-rock album also grappling with the imminent fall of the Soviet Union, full of song titles like “Don’t Shoot!” and “Revolution”. [Spotify] [iTunes] [Youtube] [English lyrics]

Sources & Further reading

Leningrad rock in general
- https://www.rbth.com/arts/2013/08/24/how_soviet_underground_music_rocked_perestroika_29179.html
- https://www.calvertjournal.com/features/show/6563/igor-mukhin-leningrad-photos-perestroika-viktor-tsoi
- http://soviethistory.msu.edu/1985-2/the-leningrad-rock-scene/
Individual bands
- Аквариум [Aquarium]
  - http://www.nytimes.com/1987/04/09/arts/for-soviet-rock-musicians-glasnost-is-angst.html
- Кино [Kino]
  - https://www.rbth.com/arts/2014/06/20/viktor_tsoi_the_last_hero_of_russian_rock_37605.html
- Ноль [Nol]
  - https://therussianreader.com/2014/12/05/lenin-street/
Key events
https://en.wikipedia.org/wiki/Tbilisi_Rock_Festival_(1980)
http://popkult.org/leningrad-rock-club/

Reading List - 2016

2017-01-03T00:00:00+00:00

[Originally posted on Facebook]

Continuing the tradition, here’s what I read last year:

★★★★★

The Wind-Up-Bird Chronicle – Haruki Murakami (tr. Rubin)

It’s almost two separate stories: a romantic drama about a fragile relationship and a gripping adventure through a surreal world. Either of them would make a good novel in itself, but Murakami’s brilliance lies in how perfectly he weaves the two together. This is the kind of novel where half the fun comes from figuring out the connections and the twisted logic underlying it all (a feeling I also had reading House of Leaves last year). [thanks Lily]

War and Peace – Leo Tolstoy (tr. Pevear & Volokhonsky)

Part sprawling epic novel, part meditation on the nature of history. Tolstoy gets inside his characters’ heads so well that by the end of the book, I felt like they were old friends. All throughout, and particularly in the “war” sections, he argues against the Great Man theory of history and proposes a bottom-up view of historical events. The resulting work is long and messy and arguably not even a “novel” at all, but at the same time it’s perfect for what it is. Isaac Babel said, “If the world could write by itself, it would write like Tolstoy,” and after reading War and Peace I couldn’t agree more. [thanks Asali]

★★★★½

Travels with Herodotus – Ryszard Kapuściński (tr. Glowczewska)

This is certainly an unusual concept for a book. Half of it is Kapuściński’s account of his journeys through India, China, and Sudan as a bright-eyed young Polish correspondent. In parallel, he recounts stories from Herodotus’s histories that his journeys remind him of. It’s remarkable how believable the characters in his retellings are: in the end, perhaps we’re not so different from the ancient Persians and Greeks. [thanks Lily]

The New And Improved Romie Futch – Julia Elliott

So, imagine “Flowers for Algernon” meets Moby-Dick. Now imagine it in a setting that’s halfway between Southern Gothic and Saunders-esque playful dystopia. And throw in surreal taxidermy, giant pigs, and Monsanto. Romie Futch is all this and more.

The Proper Study of Mankind – Isaiah Berlin

No book on this list has affected the way I think as profoundly as this collection of essays. The biggest takeaway for me was his idea that there cannot be one single ideal to strive for (“value pluralism”). He traces this idea through history, from Vico to Herder, and along the way he presents fresh perspectives on Machiavelli and Tolstoy. Other highlights include his portraits of his Russian contemporaries and his argument against “scientific history”.

Collected Stories – Stefan Zweig (tr. Bell)

I can’t believe that I hadn’t heard of Zweig before I got this book! These are 22 of the best short stories I’ve ever read, all set against a beautifully-written European landscape. The prose is gorgeous even in translation, and the characters display an immense amount of humanity. Highlights: “The Miracles of Life”, “The Governess”, “Compulsion”, “Fantastic Night”, “Mendel the Bibliophile”. [thanks dad]

Excellent Sheep – William Deresiewicz [thanks Asali]

In this book-length adaptation of his 2008 essay “The Disadvantages of an Elite Education”, Deresiewicz argues that the Ivy Leagues (and schools like them) are producing an educated elite out of touch with the rest of the country. He forms his argument from both a detailed historical account of the American education system and hundreds of firsthand accounts from students. Unfortunately, he doesn’t offer much in the way of solutions (aside from the usual clichés), but it’s a timely critique. [thanks Asali]

The Book of Tea – Okakura Kakuzō

Okakura presents a passionate defense of what he calls “tea-ism”, the philosophy of reveling in the aesthetics of the simple things in life. The book itself is an aesthetic marvel. Every paragraph is beautifully written, all the more remarkable given that Okakura learned English late in life.

★★★★

Postmodernism for Beginners – Jim Powell and Joe Lee

Corny title aside, the “for Beginners” series has been blowing me away with its intelligence and wit. I came in thinking of “postmodernism” as just a punchline, and now I’m ordering books by Baudrillard. Powell & Lee do a fantastic job of distilling key ideas of postmodern writers and artists and giving them context. [thanks Mel]

Y.T. – Alexei Nikitin (tr. Jackson)

A fascinating novella about a group of bored Ukrainian physics students who start a geopolitical role-playing game, and the consequences that follow. Nikitin’s portrayal of Ukraine in the 80s and the 2000s is by turns bleak and satirical, reminding me somewhat of Dovlatov.

Alice in Bed – Susan Sontag

How do you make a play about somebody who spent her entire life in bed? Not the easiest premise to work with, but Sontag does a masterful job. There’s not really an overarching plot, but rather a series of vignettes, ranging from mundane family moments to a Lewis Carroll-esque tea party to a bizarre robbery scene. I would love to see a production of this play. I can only imagine what it looks like live.

The Doomed City – Arkady and Boris Strugatsky (tr. Bromfield)

I’ve been waiting for this translation for literally a decade. (Seriously, I was trying to translate it myself back in high school!) The premise is typical Strugatsky weirdness: a group of “volunteers” from throughout the 20th century find themselves in a city on an alien world for an “Experiment”. Challenges present themselves, some Lynchian (a mysterious red brick house travels around abducting citizens), others political (a fascist coup overthrows the government), all while a group of “Mentors” watch from the sidelines. This is the Strugatskys at their most political, challenging the Soviet experiment overtly rather than metaphorically, and it’s easy to see why it wasn’t published until after perestroika.

The Stranger – Albert Camus (tr. Gilbert)

“Mother died today. Or maybe yesterday, I don’t know.” L’Étranger has a structure reminiscent of Crime and Punishment (part 1 is the crime, part 2 is the punishment), but unlike Raskolnikov, Meursault isn’t a philosopher. He doesn’t want to be the next Napoleon. He simply doesn’t care: not about his society, not about the people around him, not about the necessity of crying at his mother’s funeral. It’s not easy to convey Meursault’s perspective while still making him a sympathetic character, but Camus somehow accomplishes this.

101 Things I Learned in Architecture School – Matthew Frederick

Another book with a cheesy title that taught me much more than I expected. Frederick cleverly illustrates concepts such as positive/negative space, parti, and denial/reward. While the lessons are all nominally about architecture, a lot of the core concepts seem to be applicable to, for example, music, as well.

Soul of a New Machine – Tracy Kidder

The most surprising part of reading Soul of a New Machine was how little things have changed since 1981. Kidder’s depiction of Data General employees, their frustrations and motivations, is spot-on my experience at medium-to-large startups. Some of the characters reminded me so much of former coworkers that it felt uncanny. The more technical passages are sometimes dull (Kidder wrote it for an audience that understood very little about computers), but it’s still a worthwhile read.

The Word for World is Forest – Ursula Le Guin

This is Le Guin at her least subtle. The characters tend to be one-dimensional and the plot doesn’t offer many surprises. But as a hard-hitting anti-colonial polemic, it certainly delivers. I especially enjoyed her exploration of the effects of communication and the depiction of dreaming among the Athsheans.

★★★½

Bernie – Ted Rall

A surprisingly well-done graphic novel biography of Bernie Sanders. The section detailing the rise of the Democratic Leadership Council is a fascinating bit of history that I wasn’t aware of.

Death with Interruptions – Jose Saramago (tr. Costa)

The first half is solid, full of that gentle yet piercing satire that I like in Saramago. The second half just seemed to drag on without ever reaching anything satisfactory.

City of Tiny Lights – Patrick Neale

A neat little noir tale transported into modern multicultural London. Neale’s levelheaded satire of the War on Terror is surprisingly prescient for 2005.

★★★

Farewell to the God of Plague – Miriam Gross

A comprehensive, if at times dry, look into Mao’s campaigns to eradicate schistosomiasis. Gross offers a nuanced portrayal of the responses of various classes of society to the campaign, challenging both Chinese and Western conventional narratives. [thanks Asali]

★★½

Harry Potter and the Cursed Child – Jack Thorne

I’m not sure if this play was really necessary. It’s certainly fun to see this world again, but the plot is quite silly and the characters are (with a couple major exceptions) pretty weak.

Once Upon a Number – John Allen Paulos

A collection of pieces about the relationship between mathematics and stories. Some chapters are alright, but I didn’t feel like I really learned anything from it. [thanks Tikhon]

Purple State Electoral Nightmare Scenario

2016-10-22T00:00:00+00:00

See https://alexnisnevich.github.io/one-weird-electoral-trick/

In Search of a New Musical Genre, Using Spotify Data

2016-08-12T00:00:00+00:00

See https://nbviewer.jupyter.org/github/AlexNisnevich/blog/blob/master/_notebooks/exploring-music-genres.ipynb

Secrets of natural language UIs: Translating English into computer actions

2016-07-01T00:00:00+00:00

Joseph Turian and I gave a talk at Strata 2016 entitled “Secrets of natural language UIs: Translating English into computer actions”.

Here’s a video:

In the talk, we argue that natural-language UIs can enable users to interact with complicated data in a way that traditional UIs cannot. As an example, we discuss the implementation of UPSHOT’s English-to-SQL interface.

Getting a little more hands-on, I give a quick tutorial (starting around 7:20) on CCG (combinatory categorial grammar) and basic lambda calculus, finally culminating into a complete semantic parse of the example phrase “top 5 opportunities in California by amount”:

Leading up to the talk, we released montague, a little CCG semantic parsing library for Scala that emphasizes simplicity and expressibility. montague is an open-source version of the CCG parser that powered UPSHOT’s English-to-SQL translation functionality.

If you’ve ever wanted to try building a natural-language interface of your own but have been intimidated by the current NLP ecosystem, I think that montague could be a great place to start. Check out the examples package (and its documentation) to see what programs written with montague look like.

How many closed lambda-calculus terms are there of a given length?

2016-05-19T00:00:00+00:00

Update (6/2/16): Reddit user u/julesjacobs has managed to find an elegant recurrence relation for this problem! As it turns out, my approach slightly undercounted the number of terms (for length 10, I calculated 893 terms, while the true answer is 896), but was correct for N<10.

Fun fact: there are 147677899847908815123862539373715632995852464035157843195311323850864810096812226643364709183774473672545 terms of length 100.

I thought of this question a few months ago, and, as far as I can tell, nobody has really done any work on it before. And I’m not surprised – it doesn’t seem to really be a question of any practical significance, just an idle curiosity.

But hey, let’s think it though. How would we go about calculating the number of distinct closed lambda terms of length N?

Defining the problem

To start, let’s lay down some ground rules for what we mean by “distinct terms”. First off, λx.x and λy.y are clearly equivalent for our purposes – no question about that. So terms that are alpha-equivalent (in other words, equivalent to each other if bound variables are changed) should not be treated as distinct.

Also, it’s clear that we must follow some kind of conventions, or else we’d get some silly “distinct” terms: for example, we really shouldn’t treat (MN)P and (MNP) as different terms! Somewhat arbitrarily, I chose to follow the notation rules listed on Wikipedia:

Outer parentheses are always dropped: (MN) -> MN
Applications are assumed to be left associative: ((MN)P) -> MNP
The body of an abstraction extends as far right as possible: λx.(MN) -> λx.MN
A sequence of abstractions is contracted: λx.λy.λz.N -> λxyz.N

Finally, as far as counting length goes, λ, ., (, ), and variables count as a single character each, and we will ignore any whitespace.

Time for some counting!

So, now that we’ve fully stated what it is we’re counting, let’s give it a shot.

The shortest possible closed lambda term is λa.a, so there are none of length 3 or less, and only one distinct term of length 4.

How about length 5? We only have space for a single λ, and we can either take one or two variables. In the former case, we have λa.aa, and in the latter case, we have λab.a and λab.b. So, there are 3 terms of length 5.

How about length 6? It’s starting to get harder to count them all, but some work gives us 8 in total:

λa.aaa,
λab.aa,
λab.ab,
λab.ba,
λab.bb,
λabc.a,
λabc.b,
λabc.c

And so on. As long as we stick to terms with a single λ and no parentheses, it doesn’t actually seem too hard to quantify the number of distinct terms (for example, for length 6 there are 1^3 + 2^2 + 3^1 = 8 and for length 7 there are 1^4 + 2^3 + 3^2 + 4^1 = 22 such terms).

But things get tricky once we get to length 8, because suddenly we’re faced with terms that look like λa.a(aa) and λa.aλb.a. Eep, accounting for these definitely makes our task a lot trickier.

Eventually I gave up on trying to find a closed form that accounts for all of these cases, and decided to take the computational approach.

The computational approach

First I wrote a find_patterns function that finds all possible “λ patterns” up to a given length (where a “λ pattern” is a λ term with all variable slots instead occupied by the symbol X.

It works by alternating Search and Simplify steps until the results stabilize. In the Search step, patterns of length N are created through abstraction (creating λX.[term] for every term of length N-3) or through application (combining terms M and N into (MN) and (NM)).

In the Simplify step, the notation rules listed above are applied to all candidate patterns, potentially reducing their lengths.

Here’s what the function looks like in Ruby:

def find_patterns(max_n)
  patterns = Hash.new { |h, k| h[k] = [] }
  patterns[1] = ["X"] # in "patterns", X is a placeholder for variables

  # Alternate between search and simplify steps many times 
  # to build up the set of candidate patterns
  (max_n / 2 + 1).times do
    # Search
    (2..(max_n + 2)).each do |n|  # we need to go up to max_n + 2 
                                  # to get accurate patterns up to max_n
      if n > 3
        # Building up new terms through abstraction
        patterns[n] += patterns[n-3].map {|p| "λX.#{p}"}

        # Building up new terms through application
        (1...(n-2)).each {|j|
          k = n - j - 2
          patterns[j].each {|p1|
            patterns[k].each {|p2|
              patterns[n] << "(#{p1}#{p2})"
              patterns[n] << "(#{p2}#{p1})"
            }
          }
        }
      end

      patterns[n].uniq!
    end

    # Simplify (https://en.wikipedia.org/wiki/Lambda_calculus#Notation)
    # Note: the (M N) P => M N P rule is harder to encode, 
    #       but fortunately doesn't come up in N<=10 at all!
    simplified_patterns = patterns.values.reduce(&:+)
    5.times do
      simplified_patterns.map! {|p|
        p.gsub!(/λ(X+)\.λ(X+)\./, 'λ\1\2.')
        p.gsub!(/^\((.*)\)$/, '\1') if p.gsub(/^\((.*)\)$/, '\1').balanced_parentheses?
        p.gsub!(/\.\((.*)\)/, '.\1') if p.gsub(/\.\((.*)\)/, '.\1').balanced_parentheses?
        p
      }.uniq!
    end

    # Reassign to buckets by length
    patterns = Hash.new { |h, k| h[k] = [] }
    simplified_patterns.each {|p|
      patterns[p.size] << p
    }
  end

  patterns.keep_if {|k, v| k <= max_n }

  # Filter out patterns that cannot be combinators (don't start with λ)
  patterns = Hash[patterns.map {|k, v| [k, v.keep_if {|p| p.start_with? "λ"}]}]
end

After find_patterns runs, it returns a dictionary mapping each length to a list of candidate patterns. For example, the patterns for length 8 are ["λX.X(XX)", "λX.XXXXX", "λX.XλX.X", "λXX.XXXX", "λXXX.XXX", "λXXXX.XX", "λXXXXX.X"].

Each pattern is then passed into a combinators_by_pattern method that finds all possible closed terms that follow that pattern, by first filling in the variable bounds and then filling in the remaining slots in every possible way:

def combinators_by_pattern(pattern)
  # First fill in variable bounds
  bound_variables = []
  next_bound_variable = "`"  # a - 1

  5.times do
    pattern.match(/λ(X+)\./) {|m| 
      vars = ""
      m[1].chars.each {|c| vars << next_bound_variable.next! }
      bound_variables += vars.chars

      pattern.sub!(/λ(X+)\./, "λ#{vars}.")
    }
  end

  # Then fill in the slots
  # NOTE: λ expressions inside parentheses pose a challenge for this naive
  #       implementation of "bounded variables". But fortunately it's not an issue in N<=10!
  combinators = [pattern]
  10.times do
    combinators.each do |c|
      if c.include? "X"
        idx = c.index("X")
        combinators.delete(c)
        combinators += bound_variables.select {|v| c.slice(0, idx).include? v} # has this variable appeared yet?
                                      .map {|v| c.sub("X", v)}
      end
    end
  end
  combinators
end

This little program is definitely not 100% correct – it (1) doesn’t properly handle λ expressions inside parentheses, and (2) doesn’t follow the (MN)P -> MNP simplification rule – but neither of those cases come up in the N<=10 case, so I’m reasonably confident in the results I’ve gotten up to that point.

Also, the runtime complexity is exponential, so it can’t really run for much higher N anyway!

Regardless of these flaws, I’ve put the code, along with some results, up on GitHub, for anyone who wants to play with it.

A260661

The folks at the Online Encyclopedia of Integer Sequences have been kind enough to accept my humble sequence of values for 1 <= N <= 10 as Sequence A260661!

Though the entry doesn’t have as many terms as I would have liked to calculate, I’m pretty excited about my first OEIS entry. I feel like I’ve finally “made it” in some sense as a hobbyist mathematician.

Exercises for the reader

I’ve reached the limit of what I can do with this problem, but there are still things that I’m curious about!

First and foremost, finding any accurate results at all for N > 10 would be interesting. I have some preliminary results from my script (11: 3678, 12: 16299, 13: 77108), but I’m hesitant to trust them due to the issues mentioned above.

A real breakthrough would come if anyone can find a way to accurately compute results in faster-than-exponential time. In particular, I’m not ruling out a possible closed-form solution, although it would have to be pretty complex to encompass all of the notational rules.

Happy hunting!

Reading List - 2015

2016-01-02T00:00:00+00:00

[Originally posted on Facebook]

Continuing the tradition, here are the book I read in 2015:

★★★★★

Tenth of December - George Saunders [thanks Danielle]
House of Leaves - Mark Z. Danielewski [thanks James]
The Secret History - Donna Tartt
The Handmaid’s Tale - Margaret Atwood [thanks Asali]

★★★★½

Freedom - Jonathan Franzen [thanks Asali]
The Gigantic Beard That Was Evil - Stephen Collins [thanks Danielle]
Infinite City - Rebecca Solnit [thanks Asali]
I Am a Strange Loop - Douglas Hofstadter
“The Cheater’s Guide to Love” - Junot Díaz
The Brief Wondrous Life of Oscar Wao - Junot Díaz
The Pleasures and Sorrows of Work - Alain de Botton [thanks Asali]
Summa Technologiae - Stanisław Lem

★★★★

They Shoot Horses, Don’t They? - Horace McCoy [thanks Asali]
Speak, Memory - Vladimir Nabokov [thanks Asali]
The Discovery of Neptune - Morton Grosser [thanks Tikhon]
A Visit From the Goon Squad - Jennifer Egan [thanks Asali]
Marx’s Capital Illustrated - David Smith and Phil Evans
Men Explain Things to Me - Rebecca Solnit
The Dead Mountaineer’s Inn - Arkady and Boris Strugatsky
Everyman - Philip Roth [thanks mom & dad]
Doing Good Better - William MacAskill
Arabs & Israel for Beginners - Ron David

★★★½

Gentlemen of the Road - Michael Chabon

★★★

A Portrait of the Artist as a Young Man - James Joyce
Geek Sublime - Vikram Chandra [thanks Danielle]
New Urbanism and American Planning - Emily Talen

★★½

Iterating Grace - Anonymous

I’_ve also started subscribing to a few magazines this year - Nautilus and Two Lines Press - and both have been incredible reads so far. I read an issue of Lapham’s Quarterly (VIII.3 - “Philanthropy”) that had a huge influence on me, and I wish I had the time to subscribe to it too, but it’s such a dense magazine that reading it four times a year would leave no time for books 🙁

Postmortem - Asshole Transit Bureaucrat 2015

2015-10-18T00:00:00+00:00

-><-

This postmortem’s been a long time coming, in part because I don’t know if I can call my team’s submission to this last Ludum Dare, Asshole Transit Bureaucrat 2015, a game, per se. But we did end up scoring well in a few categories, so perhaps I was a bit too harsh. Let’s discuss what went right and wrong.

What went wrong

Too much conceptual complexity

This is a bit of a common problem with me and Ludum Dare: it’s been an issue with It’s Not Easy Being Muammar and, to some extent, Asteroid Tycoon. In this case, the complexity bit us in two ways.

On the one hand, it was hard for players to understand what was going on. The goal was already a difficult one to wrap your head around (make a transit system less efficient? what does that mean?), but the fact that the only thing you were able to control was the order of stops in a bus route made it even worse. Add to that the fact that there were some subtle issues with the buses that prevented them from always following their prescribed routes, and players were left completely clueless.

On the other hand, it was exceptionally difficult for us to actually make the damn thing work. We ended up only being able to spend maybe 20% of our time working on gameplay because just getting the traffic simulation part working took so much time and energy. It definitely shows.

Homogenous team

Last time around, we had a healthy mix of programmers, designers, artists, and musicians. This time, we just had three programmers and a part-time musician. To make matters worse, we were all the same kind of programmer, more or less – all functional programming nerds – and so we all tended to approach problems the same way.

This experience has really hammered into me the importance of a diverse team: it would have been great to have some people on our team looking at problems from the player’s perspective (something the three of us didn’t do nearly enough of).

Poor task management

Tying into the above, because Greg, Tikhon, and I all have similar interests, we all wanted to work on similar problems. And none of us is particularly good at being a taskmaster. As a result, task management was a constant issue.

One of us spent a full day working on a level editor, just because it was a fun problem, even though we couldn’t even integrate it into the final game. We kept ironing out theoretical edge cases in the engine because it was technically interesting, even though we had no working game to plug it into yet.

When we did split up tasks (one of us worked on the engine, another on the game UI), we didn’t sync up about the interface we wanted well enough, with the result being that integrating all our individual work together was a minor nightmare.

All of this could have been avoided if we’d been a little more principled about how we approached tasks. In the past, Jordan and I were usually the once to figure out how to manage the team. Maybe I need to start learning how to do that on my own.

What went right

Sticking to it

As the last few hours of the jam approached, it was clear that, no matter what we did, we wouldn’t have a finished game. At best we’d have three levels that kinda, sorta worked, a hastily-put-together and not quite complete interface, and an abrupt ending.

The big question wasn’t how to finish the game. It was whether to even submit it or just not bother.

I was unsure initially, but now I’m glad we submitted it, incomplete though it was. We got a ton of interesting feedback, and ended up scoring #126 in Audio and #127 in Innovation. It’s not our best yet, but hey - it’s better than nothing!

The best possible aesthetics given our constraints

The look and feel of the game are something I’m proud of, especially given what we managed to accomplish with no artists or full-time musicians.

We made some good tradeoffs. For art, rather than taking the easy way out and making a purely text-oriented game (as I’ve done before), we painstakingly put together an almost-hypnotic minimalist traffic simulator. It’s a lot of fun to look at, gameplay aside.

And for music, we only had a few hours of our Matt’s time, so we had a choice. He could either make a couple really quick tracks, or a single complex and well-put-together track. We opted for the latter, and it was a good choice - it’s just one track that repeats over and over, but it’s a damn good one, and fits well with the aesthetics of the game.

A sense of playfulness

Asshole Transit Bureaucrat 2015 was never meant to be a game that takes itself seriously. We address a topical issue, but the whole game, from the premise to the in-game text, is completely and knowingly absurd.

Not taking ourselves seriously helped us stay motivated when things weren’t going well, and I think it helped our players go easy on us, too.

What went OK

Using Elm

Well, this is a tricky one. Elm is my favorite programming language right now, and I was really looking forward to finally getting to use it in a Ludum Dare. But, as much as I enjoy working with it, I’m not sure whether it actually made our lives any easier (as compared to using JavaScript, which we’ve always done before).

The main problem with using a lesser-known language in a hackathon is the lack of frameworks and libraries that can simplify your task. In past Ludum Dares, I’ve managed to offload a lot of the complexity of my games onto libraries, whether it was rot.js for 10 Second Roguelike, javascript-astar for the pathfinding in Asteroid Tycoon, or PhysicsJS for Shattered Worlds. With Asshole Transit Bureaucrat 2015, we had to write everything from scratch (on top of the rather poorly-documented elm-graph).

On the other hand, it’s not like there’s a lot of traffic simulation libraries in JavaScript anyway, so maybe the issue wasn’t so much our choice of language as our unusual concept.

The FRP (functional reactive programming) paradigm neither helped nor hurt us, I think. In some ways, it made it easier to reason about the motion of vehicles to think of movement as a single function from State to State, but we could probably have gotten away with the object-oriented approach of treating each car as an a separate agent. It’s mostly a matter of preference.

One thing I’d like to do differently next time is to prioritize the game over the language – in other words, use the language that makes the most sense for whatever the game concept is, be it Elm, JavaScript, or maybe something else entirely.

In conclusion

All in all, this one didn’t go so well. But at least we learned something.

Next time, hopefully we’ll be able to make a complete game again.

The Impact of Donations to Bernie Sanders

2015-10-08T00:00:00+00:00

See http://alexnisnevich.github.io/bernie-donations/

Reading List - 2014

2015-01-01T00:00:00+00:00

[Originally posted on Facebook]

I got back into reading fiction in 2014. Here’s what I read last year (as best I can remember):

★★★★★

The Amazing Adventures of Kavalier & Clay - Michael Chabon [thanks Danielle]
Blindness - Jose Saramago [thanks mom & dad]
What We Talk About When We Talk About Love - Raymond Carver [thanks Danielle]

★★★★½

The Crying of Lot 49 - Thomas Pynchon
Hard to Be a God - Arkady and Boris Strugatsky (new Olena Bormashenko translation)
The Diamond Age - Neal Stephenson
Definitely Maybe - Arkady and Boris Strugatsky (new unexpurgated edition)
A Heartbreaking Work of Staggering Genius - Dave Eggers
American Pastoral - Philip Roth [thanks mom & dad]
Colorblind - Tim Wise [thanks Mel]
Cain - Jose Saramago [thanks mom & dad]
The Magician’s Land - Lev Grossman
“The Death of Ivan Ilyich” - Leo Tolstoy
Goodbye, Columbus - Philip Roth
The Search for Heinrich Schlögel - Martha Baillie [thanks Elise]

★★★★

Omon Ra - Victor Pelevin
Einstein’s Dreams - Alan Lightman [thanks Natasha]
A Dance With Dragons - George R R Martin
Travesties - Tom Stoppard
Grendel - John Gardner
The Ocean at the End of the Lane - Neil Gaiman [thanks Danielle]
Bro - Vladimir Sorokin [thanks mom & dad]
Dubliners - James Joyce

★★★½

Absurdistan - Gary Shteyngart
The Box Man - Kobo Abe
Is That a Fish in Your Ear? - David Bellos
Ice - Vladimir Sorokin
Seeing - Jose Saramago

★★★

In the Beginning … Was the Command Line - Neal Stephenson
23,000 - Vladimir Sorokin

JSON Requests with rust-http

2014-10-23T00:00:00+00:00

I’ve been playing around with Rust a lot the past couple weeks. It’s a neat little language with some cool ideas (I particularly like its memory-safety features, though they take some getting used to), but sadly the ecosystem has some catching up to do.

In particular, making HTTP requests in Rust is a bit of a mess right now. The closest Rust has to a working requests library at the moment is rust-http, which is not being maintained while its creator works on its forthcoming replacement, Teepee. Unfortunately, rust-http’s API is not the clearest to use and its documentation is rather sparse.

The fun part, though, is that this means I’ve had to figure out a lot of its quirks myself.

I’ve been working on writing a client for a JSON API and I needed to use rust-http to make lots of JSON requests and then parse their responses as JSON, with as convenient an interface as possible. Here’s what I came up with.

(Disclaimer: I have no idea whether what follows is idiomatic Rust – I’ve been using Rust for less than a month – but it works for me.)

Individual requests are encapsulated in a Request struct (method is a Method and headers is an instance of HeaderCollection):

struct Request {
  method: method::Method,
  path: String,
  headers: request::HeaderCollection,
  body: Json
}

The json_request method takes a Request struct and tries to make a JSON request with it, returning Ok(Some(response: Json)) if the response is valid JSON, Ok(None) if the response is not valid JSON, or Err(error) if the request failed in some way or did not receive a 200 OK response:

// Makes a JSON request with headers and body and expects a JSON response
// Returns one of:
//    Ok(Some(response)) : successfully received JSON response
//    Ok(None)           : response is not valid JSON
//    Err(error)         : did not receive a 200 OK response
fn json_request(request_params: Request) -> Result<Option<Json>, String> {
  let application_json = MediaType::new("application".to_string(), "json".to_string(), vec![]);

  let url = Url::parse(request_params.path.as_slice()).ok().expect("path is not a valid URL");

  let body_str = request_params.body.to_string();
  let data: &[u8] = body_str.as_bytes();

  let mut request: RequestWriter = RequestWriter::new(request_params.method, url).unwrap();
  request.headers.content_length = Some(data.len());
  request.headers.content_type = Some(application_json);
  request.headers.accept = Some("*/*".to_string());
  request.headers.user_agent = Some("Rust".to_string()); // (some APIs require a User-Agent)
  for header in request_params.headers.iter() {
    request.headers.insert(header);
  }
  request.write(data);

  let mut response = request.read_response().ok().expect("Failed to send request");
  let response_text: String = response.read_to_string().ok().expect("Failed to read response");
  // println!("{}: {}", response.status, response_text);

  match response.status {
    status::Ok => Ok(json::from_str(response_text.as_slice()).ok()),
    _          => Err(response_text)
  }
}

Here’s an example of how this method can be used:

// see http://doc.rust-lang.org/serialize/json/ to learn how to serialize datatypes into Json
// or pass your body as a raw Json string this way:
let empty_body = from_str::<Json>("{}").unwrap();

// to learn how to add custom headers to a HeaderCollection, see:
// http://www.rust-ci.org/chris-morgan/rust-http/doc/http/headers/request/struct.HeaderCollection.html
let empty_headers = request::HeaderCollection::new();

// available methods: http://www.rust-ci.org/chris-morgan/rust-http/doc/http/method/enum.Method.html
let get = method::Get;

let request = Request {
  method: get,
  path: "https://api.github.com/users/AlexNisnevich".to_string(),
  headers: empty_headers,
  body: empty_body
};

match json_request(request) {
  Ok(Some(response)) => println!("{}", response.to_pretty_str()),
  Ok(None)           => println!("JSON parsing error"),
  Err(error)         => println!("Request failed: {}", error)
}

And yeah, that’s about it. It’s still nowhere near as nice as, say, Python’s requests library, but it certainly beats working directly with rust-http’s low-level interface.

The whole thing is on GitHub for your perusal.

Some Words on Untrusted

2014-10-17T00:00:00+00:00

It’s been about six months since Greg and I released Untrusted into the world, so I figure now’s as good a time as any to write something about how Untrusted came to be.

It All Started With a Hackathon

Greg and I had wanted to make a game for a long time, and when we decided to participate in the spring 2013 CSUA hackathon at Berkeley, we figured it was as good a time as any.

Greg had an idea for a game that would take place in a Unix filesystem, where gameplay would involve moving around the filesystem and doing different operations, and where the final level would be unbeatable unless you decompile the game and rewrite part of it. It was an interesting idea, and I’d still like to perhaps try it sometime, but we couldn’t really come up with how we would implement it. I really liked the last part of the idea, though – the bit about having to modify the code of the game to make it beatable. When we gave up on the original idea, I proposed making a game where the central idea is the the player is able to modify the underlying game code. Greg thought that sounded pretty cool, so we got to work.

Design

The first thing we had to settle was how broad the scope of this “hacking” mechanic should be. If the player had arbitrary access to all game code, that would (a) be a nightmare to get working correctly, and (b) give the player far too much power to really make it possible for the game to be much a challenge. We decided to start by letting the player be able to modify the code that initializes each level, and see where we could go from there. We were considering given the player progressively more and more power to edit different aspects of the game, but we couldn’t quite figure out how that would work (though I did later use that idea in the last level of the game, 21_endOfTheLine).

Ok, so if the player could modify each level, what would the levels themselves be like. We decided to go with a roguelike style, because it would mean that we wouldn’t have to worry about graphics (though a surprising amount of work went into finding appropriate Unicode characters for things), the level maps could be represented pretty cleanly in code, there wouldn’t be too many moving parts, and, honestly, we both just really like roguelikes.

One final design hurdle was how to further limit the player’s control over the code and not make all the puzzles too trivial. Since being able to edit all of the code to a level would be too much power, we decided to only make certain lines editable. Then, as an additional level of control, we added our validator mechanism (in retrospect, we probably could have done some more interesting puzzles involving validators, but mostly we’ve just used them for basic things like ensuring players don’t create extra exits).

With these pieces in place, we had the bulk of our design ready.

Implementation

The first steps we took in our implementation was creating the map, making it be able to load a level definition, and making the player moveable. We used the rather neat rot.js library, which provided some nice abstractions, though we didn’t end up using all that many features from it. After a couple hours of work, we had a Map class that could place objects though a simple API, as well as an interactive player object. It was time to move on to the meat of the game: the evaluator.

Actually, the evaluation part is fairly simple: we just call eval. Yup. We were considering building our own parser, but that would have taken way too much work for the hackathon and eval worked well enough for our needs that we never changed it afterwards. Of course, we do some interesting things around it: we first evaluate the player-provided startLevel method on a dummy map (that supports the map API but has no in-game effect and is not visible to the player) and run validators on it. We also introduced some basic security measures here and there, like making sure certain words couldn’t be used (this could be easily circumvented though – I go into some detail later about how we designed a more robust security system).

Now that we had a map that could be initialized based on user-provided code, the final piece of the puzzle was creating the editor. This actually turned out to be the hardest part of the hackathon for us. We used CodeMirror 2, which supported almost all of the features we wanted but was missing one big one that we had to do ourselves: uneditable lines and sections. Conceptually, this didn’t seem too bad, but in between figuring out all the CodeMirror events we needed to handle and doing the right bookkeeping at all times (so that the editable areas would never move to where they shouldn’t be) ended up being pretty tricky. We ended up having to make some compromises due to time pressure: for example, in the hackathon version of Untrusted, only one line can be edited at a time and you can’t create new lines, so if you want to write multiple lines you have to go through the annoying process of navigating to each new line yourself. True multi-line editing was such a difficult feature to get working right with respect to uneditable areas that it took another year before it was properly implemented in Untrusted.

We had to cut some corners, but we finally had a working game engine. The rest of the time was mostly spent coming up with levels. I’ll admit that we were a bit uncreative (perhaps our creative juices were burned out by the amount of work it took to get the editor working), and so the first four levels we came up with all have a similar feel: you are surrounded by blocks and need to find a way to get through them to the exit, in the face of increasingly tricky constraints (these are, with some rearrangement and tweaking, still the first four levels of Untrusted).

In an effort to inject a little bit of creativity into our game, Greg and I took some time off from coding and tried to each come up with a unique premise for an interesting level. I came up with an idea for a level with invisible traps that can be identified by coloring them (this became 05_minesweeper), while Greg came up with an idea for level where the player had to bind a function call to a keystroke to get through an area (this became the basis for the function phone and 08_intoTheWoods). We only had a couple of minutes to demo our project, so at the last minute I added a cheat key to skip between levels, so that we could present only the most interesting levels when we did our demo.

End Result

Here it is, in all of its rough hackathony glory.

It seems a bit embarrassingly unpolished when I look back on it now, but the hackathon judges really liked our idea, and Greg and I ended up winning first place (and some swanky 27” monitors). Given the surprisingly positive reception we got, we decided to continue working on Untrusted until we felt it was complete enough to release.

We certainly didn’t expect to work on it for another 14 months, though.

The Next Year

Over the next year, we worked on Untrusted at an unpredictable pace, sometimes getting an enormous amount of new content done quickly and sometimes taking long breaks as our real lives caught up to us (for most of this time, Greg was working at Meraki, while I was doing my Masters at Berkeley).

Our first round of work on Untrusted, in the first few months after the hackathon, was primarily devoted to cleaning up our ugly hacked-together codebase, creating a build system and a better level file format (namely, our JSX format) to make our lives easier, and throwing out some ideas for new gameplay elements. Levels that we created during this time include 07_colors and 15_exceptionalCrossing.

Between late August and early October, we worked in some polish, adding help and menu panes, sound effects, and progress saving via localStorage. During this time, we came up with the idea for the climax of the game (at least difficulty-wise), the robot levels.

In November, Greg and I came up with our plan for how to organize the game. We decided to divide it into three chapters: the first would introduce the primary game mechanics, the second would stop holding the player’s hand and keep increasing the difficulty curve, and the third would break the rules we had previously established by introducing a bunch of new crazy things, culminating in a boss fight. We also came up with a barebones plot that was a bit cliche but worked for our purposes, and after trying a few different approaches to establishing the story, we decided that conveying it entirely through code comments would be a nice touch.

Now that we finally had a blueprint for the game, progress went much quicker, and by December we had more or less finalized the first two chapters of the game. In the meantime, we also created the loading screen and credits and found most of the background music that we wanted (aside: Free Music Archive is amazing).

Around New Year’s, we posted our new Untrusted prototype on Facebook for our friends to playtest. One thing that we noticed was that, even after all of the security improvements we had tried to add over the past year, people still managed to find a bunch of different exploits that enabled them to arbitrarily skip levels. Now, we definitely feel that finding ways to break the system is a big part of what Untrusted is about, and we don’t want to discourage it entirely, but if you can find a single exploit that lets you get around every level, that’s not very fun. In light of this, we came up with a mechanism to completely prevent players from accessing methods we don’t want them to access.

Aside: Private Methods in Untrusted

Our solution to this is actually kind of cool, so I wanted to go over it briefly.

Player-accessible classes – namely, map, player, and dynamicObject – contain exposed methods (that is, methods that the player is allowed to access) and unexposed methods (private methods that we want to be able to access but not expose to the player). Since JavaScript doesn’t really have a notion of visibility, we’ve had to use workarounds.

Our first approach was to simply have all unexposed methods begin with an underscore (_) and disallow that character in player-written code. This is not foolproof by any means, though, since there are a number of workarounds one could do to inject an underscore without explicitly typing one. We needed a better solution.

What we eventually came up was this wrapper function for exposed methods:

function wrapExposedMethod(f, obj) {
    return function () {
        var args = arguments;
        return __game._callUnexposedMethod(function () {
            return f.apply(obj, args);
        });
    };
};

where __game is a reference to the Game object, which has a local variable __playerCodeRunning and the following methods:

this._isPlayerCodeRunning = function () { return __playerCodeRunning; }

this._callUnexposedMethod = function(f) {
    if (__playerCodeRunning) {
        __playerCodeRunning = false;
        res = f();
        __playerCodeRunning = true;
        return res;
    } else {
        return f();
    }
};

Now, to enforce exposure of methods, all we need to do is (1) wrap all exposed methods with wrapExposedMethod and (2) check __game._isPlayerCodeRunning() at the beginning of each unexposed method. For example, the Map class could have methods like:

this.sampleExposedMethod = wrapExposedMethod(f, function ( [args] ) {
    ...
}, this);

this.sampleUnexposedMethod = function ( [args] ) {
    if (__game._isPlayerCodeRunning()) { throw 'Forbidden method call: sampleUnexposedMethod()';}
    ...
};

The neat thing about this approach is that exposed methods are still able to internally call unexposed methods with no problems, but each starting method call from the executed player code has to be an exposed method.

Anyway…

After taking some time off Untrusted to rest, we went back at it in February, and over the next two months we finished Chapter 3 and added some features our friends had requested a lot during playtesting, such as a notepad pane and the ability to share your solutions through automatic Gist saving. The most requested feature, the ability to edit multiple lines at once (that is, the ability to expand and contract multi-line blocks sections in the editor, and to delete or copy multiple lines) was implemented by our friend Dmitry (who, incidentally, also contributed a couple pieces of music). This was such a huge and difficult code contribution (due to the complicated interactions it had with the boundaries of editable sections) that we specifically credit him for “implementation of multiline editing” in our Acknowledgements section.

Finally, at the beginning of April 2014, Untrusted was complete and ready to be shown to the world.

Release & Disaster

When I posted Untrusted to Hacker News on the morning of April 7th, we decided we were aiming for maybe 10-20 upvotes, 50-100 if we were lucky.

We hit the front page almost immediately. Then we rose to 10th place. Then 5th. Then all the way to the top.

About an hour after the post, Untrusted began breaking: players were beating levels without advancing to the next level and getting stuck in loops. At first there were just a few isolated reports of this, but soon enough the game stopped working for everyone.

There were now thousands of people trying to play Untrusted and it wasn’t working. It was a disaster.

I did my best to track down the problem. It turned out that hosting Untrusted along with all of its music (a 200MB soundtrack) on my own VPS server was a terrible decision: the server hit its bandwidth cap within an hour and the hosting provider assumed that I was getting DDOSed and began rate-limiting all requests. Now, since Untrusted loads levels through AJAX calls, this meant that everyone playing the game at the time suddenly became sporadically (and soon, completely) unable to load levels. Even worse, we hadn’t considered the possibility of the AJAX level requests failing and had no built-in error handling to speak of. So, the game still acted as though the next level was loaded in these situations, causing levels to be overwritten by other levels and other bizarre bugs. Even worse than that, since level code was constantly written to localStorage, this meant that players’ game states were becoming corrupted to the point that even after this outage went away, the game would still be unplayable until they cleared localStorage.

Yeah, it was bad.

We had to act fast. For lack of a better idea, I moved everything over to GitHub Pages and set up a server-side redirect ASAP. Until the redirect started working, I had to work on damage control in Hacker News, apologizing profusely and directing people to our GH Pages URL (until I finished setting up GH Pages, I suggested that people could clone the repository themselves and play Untrusted locally – not the best thing to tell frustrated players but the best I could come up with at the time). I also temporarily special-cased some code in to detect when players’ localStorage` was messed up and fix it as unintrusively as possible.

After the situation stabilized, we began the process of moving all the music over to Amazon CloudFront. (Theoretically GitHub Pages doesn’t have any bandwidth limits, but I didn’t want to surprise them with 5TB/month.) CloudFront handled the load it needed very well but ended up being a little expensive, so we added a small donation button and tried some tricks to reduce requests (like not requesting music while the game is muted).

That night, Greg and I also rewrote the level loading system from scratch, so that this problem could never happen again. Instead of the game making AJAX requests for each level, all the levels are now packed into an array inside the minified source as part of the build process. This took a fair bit of bash wizardry and actually ended up one of the trickiest parts of coding the game (at least in my opinion – Greg’s a lot better at bash than I am).

Over the next month, Untrusted spread rapidly through word-of-mouth on Twitter, Reddit, and Facebook (in about that order). Fortunately there were no more major outages.

And Since Then

If I may brag for a moment: Six months after its release to the world, Untrusted has been played 700,000 times by 380,000 players from 196 countries (top five countries, in order: United States, Russia, China, Poland, France). The Untrusted GitHub repo has been starred nearly 2000 times and forked 400 times. It’s certainly become much more popular than either of us had expected from our half-baked last-minute hackathon idea last year.

A few articles have been written about us. Some of my favorites are:

our mention in PC Gamer’s list of 100 best free online games
Rachel Ponce’s review, “Untrusted is a game more like real world programming than you might think”
KillBot 86’s review
Games with Purpose’s review
GameSideStory’s review (in French)

What excites us the most is all the different ways we’ve seen Untrusted used. For example:

The Tech Club at Vianney Griffiths High School in St. Louis has used Untrusted to teach basic programming concepts. Greg and I ended up videochatting with the students of the Tech Club and sharing our experiences with programming.
CoderDojo Scotland has also been using Untrusted as an educational tool.
Janos Gyerik forked Untrusted and began constructing a new level set called HangoverX, including some fiendishly clever puzzles.
mikespook has created a Chinese localization of Untrusted, as well as adding mod support to the game and beginning to work on a new level set.
A few companies have reached out to us about using modified Untrusted level sets for interviews.
Someone even made an Untrusted subreddit, though it was only active for a few days.

All in all, we’ve been completely overwhelmed by the community that’s developed around our game. Thanks for all your support, guys :-)

What’s Next for Untrusted?

There are some exciting applications of Untrusted in the pipeline. A group in Colombia has been working on making a Spanish-language version of Untrusted designed to serve as an introduction to programming, and we’ve also heard of a similar development effort taking place in India. We’re pretty excited to see what will happen with these. To tell the truth, we had never actually intended Untrusted to be an educational game – our goal was just to make a fun little game for programmers. But seeing all of these different people take our silly game and use it to help people get into programming has been incredible.

As for the game itself, we’re no longer continually working on it, but we are still maintaining it, fixing bugs as they pop up and looking through pull requests we’ve received. Our main goal now is the make the game more community-driven – to this end, the last major feature we added to Untrusted was custom level support (player-submitted bonus levels become available from the main menu after when you get to the last level of the game). We haven’t gotten as many level submissions as we’d expected so far, but that’s our fault for adding this feature long after the main waves of players have died down.

Greg and I both learned a great deal from creating, releasing, and maintaining Untrusted, and we’ll certainly keep these lessons in mind as we work on our future projects, whatever they may be.

Setting up a Flask application on EC2 Ubuntu with Apache + mod_wsgi

2014-10-01T00:00:00+00:00

Last weekend I had to restart my melody.py live demo, as the EC2 instance it was on was outdated. In the process I had to figure out again how to set up Flask applications using mod_wsgi. As a relative newbie to both EC2 and Apache, the process took longer than I’d like, so I’m documenting the steps I took in detail, in part to make it easier for me next time I need to do something like this.

I’m going to break the process up into three bite-sized chunks:

Launching an EC2 instance
Setting up Flask and Apache
Configuring mod_wsgi

Prerequisites

I assume you have a working Flask app that has a remote git repo, perhaps on GitHub.

Launching an EC2 Instance

First things first, let’s set up an EC2 instance running Ubuntu. I’m going to defer to Amazon’s documentation here, which is far more complete than anything I could write here.

A few things to keep in mind:

Make sure that the security group you assign your instance to allows HTTP access (port 80) from everywhere (source 0.0.0.0/0) and allows SSH access (port 22) from your local machine’s IP.
Make sure that your instance is assigned a Public IP (this setting may be unchecked by default).

Once the instance is created, let’s make sure that we can SSH into it:

ssh -i ~/.ssh/<path to .pem file> ubuntu@<public DNS address>

If SSH complains about your private key file being insecure, chmod it to 0600 and try again:

sudo chmod 0600 <path to .pem file>

Setting up Flask and Apache

Now that the EC2 instance is up and running and we can SSH into it, let’s install Apache, mod_wsgi, Flask, and git:

sudo apt-get install apache2 apache2-base apache2-mpm-prefork apache2-utils libexpat1 ssl-cert
sudo apt-get install libapache2-mod-wsgi python-pip git
pip install flask

and restart Apache to complete the installation of mod_wsgi:

sudo service apache2 restart

For more information on this step, see the DigitalOcean tutorial.

Configuration

First things first, we need to make a WSGI file for our application. This is the file that tells Python how to communicate with a web server. I use a very barebones WSGI file:

import sys
sys.path.insert(0, '/var/www/{appname}')

from app import {app global variable in app.py} as application

(for more information about this file, see the Flask mod_wsgi documentation). For convenience, I like to store this file directly in git, so save it as [appname].wsgi in our app root directory and add it to our app’s git repo.

Now, let’s deploy this application to the server:

cd /var/www/
git clone [path to git repo] # you may need to mess with permissions on the /var/www/ directory

Now, we just need to configure Apache to use mod_wsgi and point to our WSGI file.

Go to /etc/apache2/sites-available/ and make a new file – let’s call it amazonaws.com.conf. This is the file that will tell Apache what to do when someone visits our site. A basic template that works for me is:

NameVirtualHost *:80

<VirtualHost *:80>
        ServerName [our EC2 server's public DNS domain name]
        WSGIScriptAlias / /var/www/[appname]/[appname].wsgi
        <Directory /var/www/[appname]/>
                Order allow,deny
                Allow from all
        </Directory>
        ErrorLog ${APACHE_LOG_DIR}/error.log
        LogLevel info
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Now let’s enable the site we just configured:

sudo a2ensite amazonaws.com
sudo a2dissite 000-default  # and for good measure, let's disable the default placeholder site

Almost done! Now we just need to restart apache:

sudo /etc/init.d/apache2 reload

is what most tutorials suggest, but for some reason I kept seeing the default Apache loading page after this. Without digging too deeply, I solved this problem by doing:

sudo apachectl restart

instead.

For more information on this step, see this DigitalRiver tutorial.

If you’re made it this far, hopefully your Flask application is running correctly on EC2.

Postmortem - Shattered Worlds

2014-09-27T00:00:00+00:00

[Original post on ludumdare.com]

True to my promise in my last postmortem, I worked in a team again for LD#30. We had a great mix of veterans (Alex, Greg, Natasha, Tikhon doing coding, and Jordan doing art and design) and newcomers (Beth doing level design and writing, and Aylan doing music and SFX), and we’re pretty proud with our end result: Shattered Worlds.

Let’s talk about what we did right and wrong. But words are boring, so let’s illustrate this postmortem with a selection of some of the lolcommits that some of us took during the jam.

What went right

Simplicity over complexity

Unlike last time‘s brainstorming session, where we tried to combine many ideas together into a very complex game, this time around we tried to distill our ideas down into a core gameplay element. We tried a few ideas that involved manipulating properties of different worlds before finally settling on the mechanic of overlaying levels together.

A mechanic that gave us a lot to work with

At the same time, we were fortunate to not be constricted too much by our gameplay mechanic. We were able to come up with a wide variety of settings and obstacles to go with it, from bees to bears to lasers.

Good team composition and task management

Having a team of people with diverse talents came in handy once again, as we were able to split up the work pretty well. Compared to last time, we had one more designer and a few fewer programmers. This seemed like a pretty positive change: two designers meant that a lot of thought went into every aspect of level design and also that Jordan was free to focus more time on the art assets, while we had just enough programmers that we all knew what we were doing and weren’t stepping on each others’ toes too much.

Distinctive style and flavor

We tried hard to make a game that looked and felt unique, and I feel that we did a pretty good job in that regard. Giving each world type a completely different palette and style helped (initially we were hoping to have distinctive music for each world too, but sadly our musician didn’t have quite enough time to make that happen), as did the quirky in-game messages. There are certainly places where we could have used more polish, but overall I think we didn’t do too bad in the style department.

Hint system

We always think our games are easier to figure out than they actually are, simply by virtue of playing them over and over while making them. In light of this, Tikhon had the foresight to propose a hint system to guide players when they needed help. Many of us initially thought that hints would be unnecessary and would make the game too easy, but we have since seen the error of our ways: quite a few comments specifically mentioned the hints as an important component of the game.

What went wrong

Switching frameworks in the middle of it all

For the first day of coding, we used EaselJS. Easel is ok, but none of us knew it particularly well, and it didn’t seem to have anything built-in that would help us make a platformer (all in all, we essentially used Easel as a wrapper for canvas). Our code quickly became a tangled mess of modified example code we found online and our own attempts at doing things like collision detection. It was clear that we couldn’t really go on like this.

The solution was PhysicsJS, an awesome physics simulation library that had all the tools we needed to cleanly and efficiently create the physics for our game. Unfortunately, porting our code over from Easel to PhysicsJS was not as easy as we had hoped, because the two libraries represent objects and their interactions in completely different and incompatible ways. Come Saturday evening, Tikhon began to spearhead the effort to switch to PhysicsJS while the rest of us continued working in Easel, but by Sunday we realized that the only way PhysicsJS would work for us would be if we essentially did a complete rewrite.

This was a huge decision. Basically all the code that we wrote on Saturday and early Sunday had to be scrapped, and we had to build up the game almost from scratch. As Sunday came to an end, we just barely got a couple of levels working. The switch to PhysicsJS was complete, but it left us with less than a day left to implement most of the levels. Somehow, we still managed to finish. I still think the decision that we made was the correct one — our old physics were horribly buggy and PhysicsJS enabled us to do a lot of cool things that we hadn’t even considered at first (like movable crates in the zombie level, which we got essentially by accident). If we’d stuck with Easel instead, our final game may have had a few more levels in the end, but it wouldn’t play nearly as well. Of course, the best thing we could have done in hindsight would have been to actually research our framework / library options ahead of time rather than >24 hours in.

Balancing was pretty hard

In particular, the low-gravity space levels — while very cool conceptually — proved to be an enormous pain to plan for during level design, because for every level after the first space level, we had to consider the possibility of the player activating low-gravity and try to make the level non-trivial to beat. We definitely didn’t do the best possible job here: some levels are still much too easy with the use of low-gravity.

If we’d had a little more time, we had plans for a system where some levels would be disabled from overlaying other levels, based on game objects not being able to overlap one another. This feature would have conceivably enabled us to make more levels without having to necessarily consider every possible pair of overlapping levels.

We couldn’t get the controls quite right

We spent some time making the game feel right as a platformer, and even had some of our platformer-loving friends playtest it from time to time, but we never got the feel quite right, not even within PhysicsJS. Fortunately, this doesn’t seem to bother too many people, though aforementioned platformer-loving friends were a bit unhappy with it.

Not much of an ending

You go through 11 levels accompassing four different worlds and fight through everything from space battles to a zombie siege, and when you reach the end you get … a dialog essentially saying “The End”. It would have been great to have a real ending of some kind, but by the last hour of the jam we unfortunately still had our hands full just making the rest of the game work. Alas.

All in all

We’re pretty happy with how Shattered Worlds turned out, and LD#30 was a great experience overall. It seems to me that we’re improving as game creators and as a team (well, it’s not exactly the same team, but it’s close enough), though we’ll have to wait for judging to end to see how true that is. Hopefully we can make participating in the LD Jam a regular thing. :-)

A First Foray into Experiential Art

2014-06-27T00:00:00+00:00

I presented “A Snack in Every Port” at Appetite Obscure #8: Seacret, an interactive potluck.

From the description on the site:

A Snack in Every Port

A map and a compass lie in front of you. Look for the dot that represents your ship. Place the point of the compass on the dot, and mark the ship’s next location with the pencil. Where you land determines what you eat.

Open Water: Crust of Bread & Slice of Lime
Cookie Confederation: Sugar Cookie
Appetizer Archipelago: Mini Shishkabob
Cupcake Coast: Chocolate Cupcake
Sultanate of Salmon: Cracker with Salmon and Cream Cheese
Principality of Parch: Dangerous-looking Bottle of Unidentified Booze

Appetite Obscure was great fun altogether (from preparing the dish to demonstrating it to trying other dishes), and I look forward to attending more interactive potlucks in the future.

Postmortem - Asteroid Tycoon

2014-05-07T00:00:00+00:00

[Original post on ludumdare.com]

For this Ludum Dare, I worked with a group of friends to create Asteroid Tycoon. I’ve done the Compo a few times in the past, but none of us had ever game jammed in a team before, and it was quite the experience. I’ve learned that doing Ludum Dare with a team brings with it both enormous benefits and unexpected challenges.

What went right

Diverse talent

Being able to work with an artist, a UI designer, a game designer, an amateur musician, etc (some people filled multiple of these roles) was an amazing experience compared to previous Ludum Dares. For the first time, I’ve submitted a game that actually looks and feels really solid.

Combining multiple ideas together

During our team brainstorming session, a few ideas became big hits: a candy-box-like about building an asteroid mining empire, something involving moles and tunneling, and a game with programmable robots. The idea for Asteroid Tycoon came about as we realized that we could combine elements from these three ideas into a coherent concept.

Constant balance tweaking

Gameplay elements underwent many changes as work on the game went on. For example, the robot upgrade progression initially was triggered by amount of minerals collected (a different mineral for each robot), but switching to a depth-based system proved to be a lot more enjoyable for the player. For much of the weekend (at least once we had something playable), at least one person (usually a team member taking a break) was playing the game at all times, so we constantly had feedback that we used to adjust parameters, and in many cases, rework entire mechanics.

Cute little touches

Little things like the marquee seem to have made a big difference in enjoyment and immersion. Also, some of our last-minute graphical tweaks, such as the beaming-down animation from the ship, improved the look of the game rather significantly.

What went wrong

Repetitive music

Once we had a short loop of music, we deemed that “good enough for now” and decided to come back to the music situation if we had time. Unfortunately, we didn’t have any more time to work on music, and the result was slightly disastrous, as the constantly repeating main loop proved to be annoying to many players. We quickly added a mute button, but the damage was already done. In the future, I’d hesitate to add music unless it was sufficiently varied to not be a hindrance.

Not enough organization

Never having worked with a team on Ludum Dare before meant that my default workflow was an informal, poorly defined task list and a single development branch in git. This is reasonable enough for one person, but proved to be a disaster when working with a team of 7 people. On several occasions, lack of communication led to two programmers separately implementing the exact same feature or our two artists stepping on each others’ toes. Meanwhile, all work being on a single branch meant that nearly every commit resulted in a merge conflict, which proved to be very time-consuming in the long run. In retrospect, we should have had a more well-defined task-assignment scheme and used feature branches.

Gameplay not explained clearly enough

The primary gameplay mechanic in Asteroid Tycoon works as follows: you first click on a robot to build and then click on a destination square for it to try to move to. However, we didn’t explain this very clearly within the game itself, which led to many players thinking that they had to click on the surface or on the spaceship — the result being that the robots reached their destination immediately and from then on proceeded to move in a way that would have appeared to be more or less random to the players.

To make matters worse, important information is conveyed to the player via printouts that regularly appear but go away on the next mouse-click. What we didn’t anticipate was that most players click around so quickly that they don’t even get a chance to see the printout before accidentally closing it (old printouts are still saved in the top-left panel, but most players never bothered to use it).

In retrospect, we should have done a better job of explaining exactly how gameplay works from the start, and come up with a different way to hide printouts (perhaps by giving them a Close button). These are both hopefully things that we will work on in the post-Jam version.

Only starting work after 18 hours

We made the conscious decision to not start work until noon (PST) Saturday, to give our brains plenty of time to process our different ideas. While this gave us lots of brainstorming time, it put us at a disadvantage time-wise compared to most Jam teams, and led to us being very pressed for time near the end (compounded by the fact that many of us had to go to work on Monday). In the future, I’d like to spend a little less time brainstorming and perhaps at least try to get a little bit of actual coding done Friday night.

All in all

After trying a Ludum Dare with a team for the first time, I don’t know if I can go back to doing the Compo! Not only were we able to build something bigger and more polished than any one of us could do on their own, but we had lots of fun working as a group. We’ll try to work together in future Ludum Dares.

As far as Asteroid Tycoon goes, we’re really happy with how it turned out. We’ll probably tweak it a little bit post-Jam — primarily to make the instructions more clear, make the music a little less repetitive, and fix some sneaky bugs that people have reported.

Postmortem - 10 Second Roguelike

2013-09-03T00:00:00+00:00

[Original post on ludumdare.com]

Well, I've never done one of these before, so here goes nothing.

What went right

Long brainstorming session

I was fortunate in having a group of friends help me brainstorm ideas when the theme was first announced. About a dozen ideas were thrown around, some interesting and some worthless, until I finally settled on this one. My first few ideas were tempting but also ultimately crappy, so being able to bounce back ideas with other people saved me from falling into the usual trap of doing the first thing that fell into my head.

Playing to my strengths

This isn't the first roguelikelike I've made, and for good reason, aside from loving the genre: I'm not any good at art, at all. In almost any other genre, I'd be forced to spend long hours putting together terrible art, but fortunately this wasn't an issue for me here.

Not reinventing the wheel

For level generation and field-of-view computation, I used rot.js, an excellent set of utilities specifically designed for roguelikes. I probably could have managed without rot.js, but when you only have 48 hours, why not limit the work you have to do as much as possible? Using rot.js also offered the huge advantage of giving me something vaguely playable within an hour or two of starting, meaning that I could spend the rest of the competition working on making gameplay as fun as possible.

Addictive gameplay

My main objective with this game was making it as fun and as fast-paced as possible. To this end, I made death be completely harmless, let players immediately create a new character in one click and jump back in the game, and provided a silly soundtrack to accompany the dungeon romp. When my playtesters (er, friends) started to refuse to give me back the computer, I knew that I had stumbled onto something.

A little bit of flair

Roguelikes are inherently rather bland visually, so small touches to make the game look good were important for the player experience. Things like smooth lighting, a shiny experience bar, and a memorial to dead characters at the end of the game were all relatively quick to implement and provided a nice distraction from the big-picture stuff for me, while making the game look much more polished. Don't skimp on the small stuff!

What went wrong

No good plan for the second day

Most of the game features (timer, dungeon exploration, combat, character creation) were completed in the first 24 hours, which made me very optimistic about the next 24 hours. However, while there were a lot of things that I wanted to do (including items, spells, and more interesting monsters), I spent so much time debating about what order to do them in that I wasn't able to work most of these features into the game. In the end, my second day consisted mainly of making levels and creating the boss fight and ending, as well as doing a lot of playtesting and minor tweaking. Next time around, I'd like to budget my time a little more effectively, especially for the second day.

Not enough replay value

Well, you beat the boss, and then the game is over. Several playtesters told me the game needed to have a way to continue playing after the boss fight, but, given how messy my second day was, I declined to implement this at the time. I did eventually address this issue by adding an Infinite mode to the post-compo version though.

Dungeon annoyances

I didn't spend enough time making sure that all dungeon stairs are reachable within 10 seconds. Sometimes the stairs aren't, though they should still be reachable if you have a faster character (speed > 1.0). And I'm pretty sure there's a (small) chance of the boss appearing in an unreachable location on the final level, though I haven't personally seen that. Regardless, I should have spent more time ensuring that every game is beatable, regardless of luck of the draw.

Not enough lore

The game's story consists of two paragraphs of flavor text that I hastily cobbled together as the minutes were counting down. I suppose it's not that big a deal for a game that seems to have more in common with Quake than with Rogue, but it still would have been nice to have an actual story.

All in all

I've taken part in three Ludum Dares before, but this was by far my most successful one: I made a game that most players found to be fun, which is a feat that I have never really achieved before. Next time around, I will hopefully interpret the theme a little more creatively, and I hope to address my biggest issue from LD27: insufficient planning of secondary game features. Still, I'm pretty happy with Ten Second Roguelike. Let me know what you think!

Rolling Your Own Mechanical Turk Form with ExternalQuestion and Rails

2013-05-23T00:00:00+00:00

Motivation

For a research project that I’m working on, I made a survey that feeds data into a Rails application. I wanted to create Mechanical Turk HITs that direct users to the survey (on my site), and send the results directly to the server. I also wanted to be able to dynamically create the HITs from within the Rails application itself.

Fortunately, the Mechanical Turk API provides a means for doing this through the ExternalQuestion structure. Here’s how I accomplished this.

Creating the HIT

To create the HIT from within my application, I used the ruby-aws gem. Creating HITs with it is pretty simple:

require 'ruby-aws'

# For production:
# mturk = Amazon::WebServices::MechanicalTurkRequester.new :Host => :Production

# For testing:
mturk = Amazon::WebServices::MechanicalTurkRequester.new :Host => :Sandbox

mturk.createHIT(
  :Title => TITLE,
  :Description => DESCRIPTION,
  :MaxAssignments => NUM_ASSIGNMENTS,
  :Reward => { :Amount => REWARD_AMOUNT, :CurrencyCode => 'USD' },
  :Question => File.read QUESTION_PATH,
  :Keywords => KEYWORDS
)

Most of these parameters should be fairly straightforward. QUESTION_PATH is the relative path to a question XML file (I keep mine under /app/lib/mechanical_turk_questions/). For ExternalQuestions, the format of the XML file is very simple (the latest schema URLs can be found here ):

<?xml version="1.0" encoding="UTF-8"?>
<ExternalQuestion xmlns="[URL of schema]">
  <ExternalURL>[path to survey]</ExternalURL>
  <FrameHeight>[height of frame]</FrameHeight>
</ExternalQuestion>

Now, running the above code should create the desired number of HITs, pointing to the desired path. Of course, before we do that we should probably set up the form itself.

Setting up the Form

In order to integrate with Mechanical Turk, the form needs to:

submit results to https://www.mturk.com/mturk/externalSubmit
pass on the assignmentId parameter (I also pass on the hitId parameter, just in case)

The controller action should look like:

class SurveyController < ApplicationController
  def show
    @assignment_id = params['assignmentId']
    @hit_id = params['hitId']

    # whatever other setup is needed for the survey
  end
end

The template of the form should look like:

<% form_tag("https://www.mturk.com/mturk/externalSubmit", :method => "post") do |form| %>
	<!-- the actual survey goes here -->
	<% hidden_field_tag 'assignmentId', @assignment_id %>
    <% hidden_field_tag 'hitId', @hit_id %>
    <% submit_tag "Submit", :disabled => true, :id => 'submitButton' %>
<% end %>

To ensure that the survey is fully completed, I made the Submit button be disabed at first, and only enabled it via JavaScript once every field was filled out. This is of course strictly optional.

Submitting the Result

When a Turker hits the Submit button in the survey, I want two things to happen:

The results should be passed on to the server
Amazon should be notified that the Turker completed the survey

On the server side, things are pretty simple - I gave SurveyController a submit action that handles the incoming data:

class SurveyController < ApplicationController
  def submit
    # process the survey result
  end
end

Now comes the tricky bit - how do we get the data to both the server and to Amazon?

At first, I tried submitting the survey to my server first, then processing the data and redirecting Turkers to the Amazon URL. However, this didn’t work - the Turkers all ended up receiving a “There was a problem submitting your results” error:

As it turns out, Amazon only seems to allow externalSubmit requests that originate from the client, not ones that originate on the server or are redirected.

So, the only way to do what we want to do is to submit the form twice - once to the server and once to Amazon. This is possible with a bit of jQuery trickery:

$("input:submit").click(function(e) {
    e.preventDefault();

    // submit to our server
    $.ajax({
        url: "/submit",
        type: 'post',
        data: $('form').serialize(),
        success: function(result) {
            // submit to mechanical turk
            $('form').submit();
        }
    });
});

Now, when the submit button is clicked, the contents of the form are first serialized and passed to /submit, and only after that request is successfully made does the form itself submit to the externalSubmit URL.

In Conclusion

Putting these pieces together, we get a pretty nifty workflow:

HITs can be created from within the application.
HITs direct Turkers to the survey within the application.
Turkers fill out the survey.
The results are submitted to the application.
The results are then submitted to Mechanical Turk, to credit the Turkers for their work.

The one step that’s still missing is that each Turker’s submission needs to be manually approved or rejected. I’m currently working on figuring out how to automatically approve/reject work - perhaps I’ll write about it in a later post.

The Church-Rosser Theorem and the Consistency of Lambda Calculus

2013-05-17T00:00:00+00:00

For my Computability final paper, I wrote about the Church-Rosser Theorem and the consistency of λ-calculus, as well as introducing the general theory of λ-calculus and showing how arithmetic can be performed within it.

What is the Church-Rosser Theorem?

The main operation of λ-calculus (at least in its standard form) is β-reduction. Loosely speaking, β-reduction means that, given a function that takes an argument $x$ and returns $M$, applying this function to some argument $y$ yields the same result as substituting $y$ for every occurrence of $x$ in $M$. In the notation of λ-calculus, we write this as:

\[ (\lambda x . M) y = M [x := y] \]

where $(λx.M)$ means “the function that returns $M$ given $x$” and $M[x := y]$ means “the result of substituting $y$ for $x$ in $M$”.

A statement $M$ can be β-reduced to $N$ if you can apply the above formula some number of times to $M$ to turn it into $N$.

For example, $((λx.(λy.y))z)z$ β-reduces to $z$, because

\[ \Big(\big(\lambda x . (\lambda y . y)\big) z\Big) z = \big((\lambda y . y) [x := z]\big) z = (\lambda y . y) z = y [y := z] = z \]

The Church-Rosser Theorem for β-reduction states that whenever some expression $M$ can be β-reduced to two different expressions $N$ and $P$, there exists some expression $Q$ that both $N$ and $P$ can β-reduce to. This property is also called confluence, and is sometimes referred to as the diamond property, because the resulting graph of reductions looks like this:

Proving the Church-Rosser Theorem

In my paper, my proof of the Church-Rosser mainly follows the classic Tait-Martin-Löf proof, which is well-known as one of the simplest proofs of the theorem, relying on nothing more than a basic understanding of the theory of β-reductions.

The proof has three stages:

First, I introduce a new notion of reduction, that I call ◊, and show that ◊ satisfies the diamond property. In this section, I follow Barendregt’s exposition of the Tait-Martin-Löf proof.
Next, I show that ◊-equivalence is the transitive closure of β-equivalence. In other words, the β-equivalence relation is the minimal superset of the ◊-equivalence relation that satisfies transitivity.
Finally, I prove the Strip Lemma, which states that if a notion of reduction satisfies the diamond property, then so does its transitive closure. Thus, since ◊ satisfies the diamond property, so does β. I primarily follow Robert Pollack’s simplified proof of the Strip Lemma.

What does this mean for consistency?

In the final section of my paper, I show that it follows from the Church-Rosser Theorem that every expression in λ has a unique “β-normal form” - that is, a unique form from which the expression cannot be β-reduced any further.

Then, I show that two terms cannot be equal to each other in λ-calculus if they are not β-equivalent. This is a result that follows directly from the axioms of λ-calculus.

Putting all of these pieces together, if two terms $M$ and $N$ do not share the same β-normal form, then the statement $M = N$ is not provable in λ-calculus.

It’s easy to find two terms that do not share a β-normal form - for example, $I = λx.x$ and $K = λx.(λy.x)$. Thus, the statement $I = K$ is not provable in λ-calculus, and so λ-calculus is a consistent theory, because if λ-calculus were inconsistent, then by definition every (syntactically valid) statement would be provable in it.

Interested in this sort of thing?

If this has piqued your interest, check out my paper here.

My Experience with Steam for Linux (64-bit Linux Mint Debian Edition)

2013-03-25T00:00:00+00:00

I’ve been successfully using Steam on my 64-bit Linux Mint Debian Edition desktop for the past month, and I thought I should make a small post about how I got it working and how well various games that I’ve tried work. I’ll try to keep this post updated as I try running more games.

Installing Steam on Debian

Installing Steam on 64-bit Debian is a bit of a pain. After trying and failing to get it to work myself, I followed aspensmonster’s instructions, which go over everything from setting up multiarch to modifying and rebuilding the steam package to fix the bugs in it. The guide has been updated fairly regularly as new versions of Steam for Linux came out, and should be more or less up-to-date.

This Debian install script may be helpful as well, but I haven’t had a chance to try it.

Games

Worked immediately	Required modification	Works, but with issues	Couldn’t get to work
Aquaria	Bastion	Dungeons of Dredmor	Team Fortress 2
Closure	Faster Than Light
Frozen Synapse
Gratuitous Space Battles
VVVVVV
World of Goo

Aquaria

Aquaria works perfectly from the start.

Bastion

At first, when launching Bastion I just got a black screen after the Supergiant logo appeared. This is a known issue with S3tc textures and I could fix it by opening /.steam/root//SteamApps/common/Bastion/Linux/run_steam.sh (your path may differ) and editing the ./Bastion.bin.x86 line to be force_s3tc_enable=true ./Bastion.bin.x86.

Closure

Closure works perfectly from the start.

Dungeons of Dredmor

Dungeons of Dredmor works, but suffers from fairly noticeable lag compared to its performance on Windows. It looks like this is a known Linux problem, but I haven’t been able to find a fix yet.

Faster Than Light

Faster Than Light crashed on startup at first, due to my open-source video driver. There are two possible solutions; I opted for the easy way out and simply deleted /.steam/root/SteamApps/common/FTL\ Faster\ Than\ Light/data/amd64/lib/libstdc++.so.6 (your path may differ). Faster Than Light now runs perfectly, and, surprisingly, is actually much faster under Steam Linux than the standalone Linux binary of FTL is.

Frozen Synapse

Frozen Synapse works perfectly from the start. There seems to be a bit of mouse lag, but it’s hardly noticeable.

Gratuitous Space Battles

Gratuitous Space Battles works perfectly from the start.

Team Fortress 2

I haven’t yet been able to get Team Fortress 2 to work. When I try to launch it, I get the error "Required OpenGL extension "GL_EXT_texture_compression_s3tc" is not supported. Please install S3TC texture support.". It looks like others have had this problem, but none of the fixes that I’ve seen mentioned work for me. However, I’m not an FPS fan, so I haven’t put very much time into trying to get TF2 to work.

VVVVVV

VVVVVV works perfectly from the start.

World of Goo

World of Goo works perfectly from the start.

Face Tracking with OpenCV and a USB Missile Launcher

2013-02-19T00:00:00+00:00

Check out Sentinel for Linux, Windows, and OS X on GitHub

I recently got my roommate a Dream Cheeky Thunder USB missile launcher. It was fun to play around with for a bit, but the included software was very limited in its functionality. Just when I was ready to dismiss it as an overpriced toy, my roommate came up with a great idea: could we mount a camera to the turret and make it automatically aim and fire at faces?

After a couple weeks of experimentation, we came up with Sentinel, a Python script that does just that, making heavy use of the excellent OpenCV computer vision library. Here’s how we did it.

But first, a video demo!

Note: The above video was filmed while our computers were rather bogged down, and Sentinel usually runs signficantly faster than that (especially on Windows and OS X, where it can go as fast as 3 iterations per second).

Step Zero. The concept

The main loop of the program is conceptually quite simple:

while True:
   camera.capture()
   face_detected, x_adj, y_adj = camera.face_detect()
   camera.display()
   if face_detected:
      turret.adjust(x_adj, y_adj)
      turret.ready_aim_fire()

At each iteration, the camera takes a picture, which is then processed to detect any faces. If a face is detected, the turret adjust itself to bring the face closer to the center of the camera’s field of view. If the face is close enough to the center and the turret is armed, it then fires a missile at its target. The camera’s output is also sent to the screen, with an ominous red reticule drawn over a detected face.

Note: I’m only showing bits of pieces of our source code here, and I’m simplifying it where I can to keep the code chunks relevant. You can see the full source code here.

Step One. Assembling the hardware

We had a small Logitech C270 webcam lying around, so we taped it to the top of the turret, aiming it roughly parallel to the trajectory of the turret’s missiles, as shown in the photo above.

The webcam’s position about an inch above the turret means that when the camera is pointing toward a target’s face, the turret usually ends up shooting them in the neck or chest, which is a nice side effect, since neither of us wanted to get shot in the eye.

Step Two. Controlling the turret

Controlling USB devices from Python requires the PyUSB library, which in turn needs a low level driver - we used libusb.

After that, connecting to a USB device is just a matter of finding its vendor and product IDs. In the case of this missile launcher, lsusb reported these IDs to be 2123:1010. In Linux, you also need to detach the kernel driver if it is active:

class LauncherDriver():
   # Low level launcher driver commands
   # this code mostly taken from https://github.com/nmilford/stormLauncher
   # with bits from https://github.com/codedance/Retaliation
   def __init__(self):
      self.dev = usb.core.find(idVendor=0x2123, idProduct=0x1010)
      if self.dev is None:
         raise ValueError('Missile launcher not found.')
      if sys.platform == 'linux2' and self.dev.is_kernel_driver_active(0) is True:
         self.dev.detach_kernel_driver(0)
      self.dev.set_configuration()

Now that the device is configured, we can send commands to it. Fortunately for us, others had already discovered the commands that the device accepts to position its turret, fire missiles, and toggle its built-in LED:

   def turretUp(self):
      self.dev.ctrl_transfer(0x21,0x09,0,0,[0x02,0x02,0x00,0x00,0x00,0x00,0x00,0x00])
   def turretDown(self):
      self.dev.ctrl_transfer(0x21,0x09,0,0,[0x02,0x01,0x00,0x00,0x00,0x00,0x00,0x00])
   def turretLeft(self):
      self.dev.ctrl_transfer(0x21,0x09,0,0,[0x02,0x04,0x00,0x00,0x00,0x00,0x00,0x00])
   def turretRight(self):
      self.dev.ctrl_transfer(0x21,0x09,0,0,[0x02,0x08,0x00,0x00,0x00,0x00,0x00,0x00])
   def turretStop(self):
      self.dev.ctrl_transfer(0x21,0x09,0,0,[0x02,0x20,0x00,0x00,0x00,0x00,0x00,0x00])
   def turretFire(self):
      self.dev.ctrl_transfer(0x21,0x09,0,0,[0x02,0x10,0x00,0x00,0x00,0x00,0x00,0x00])
   def ledOn(self):
      self.dev.ctrl_transfer(0x21,0x09,0,0,[0x03,0x01,0x00,0x00,0x00,0x00,0x00,0x00])
   def ledOff(self):
      self.dev.ctrl_transfer(0x21,0x09,0,0,[0x03,0x00,0x00,0x00,0x00,0x00,0x00,0x00])

Step Three. Controlling the camera

Now that we’ve set up the rocket launcher, the next step is accessing the webcam to make it take a picture once per loop iteration. We ended up doing this in a few different ways.

Linux: streamer

On Linux, I was excited to find streamer, a fast and simple-to-use photo/video capture tool. Even after we switched to using OpenCV’s photo capture capabilities on Windows, streamer still ended up giving the best performance on Linux.

# captures a single frame - currently a platform-dependent implementation
def capture(self):
   if sys.platform == 'linux2':
      # on Linux, use streamer to generate a jpeg, then have OpenCV load it into self.current_frame

      img_file = 'capture.jpeg'
      subprocess.call("streamer -q -c /dev/video" + self.opts.camera + " -s "
            + self.opts.image_dimensions + " -b 16 -o " + img_file, stdout=FNULL, shell=True)
      self.current_frame = cv2.imread(img_file)

Windows and OS X: OpenCV

Capturing photos on Windows proved trickier. Initially, we used CommandCam, which gave good results but was a litle too slow to use. We finally switched to using OpenCV’s image capture methods, but ran into a serious problem.

The images that OpenCV was processing were always a few frames out of date compared to the images the camera was taking, which ended up making the turret oscillate back and forth instead of homing in to its target, because it was adjusting its position based on outdated information.

This behavior is actually intentional in OpenCV: images are stored in a buffer and retrieved as quickly as possible, and the latest image taken is not generally the image retrieved. Since OpenCV is generally used in a situation where continuous footage is being taken from a video camera, this is not usually a problem. However, since we were taking only one picture at a time and then repositioning the turret based on each picture, this behavior was unacceptable.

Our solution was a little hackish but succeeded in correcting the problem: we simply made a clear_buffer method that repeatedly grabs images from the buffer until only the latest image is left, slowing the process down slightly but greatly improving the turret’s behavior:

# grabs several images from buffer to attempt to clear out old images
def clear_buffer(self):
   for i in range(self.opts.buffer_size):
      if not self.webcam.retrieve(channel=0):
         raise ValueError('no more images in buffer, mate')

The webcam is set up for use with OpenCV via self.webcam = cv2.VideoCapture(int(self.opts.camera)) within the Camera class’s initializer, and frames are captured like this:

# on Windows and OS X, use OpenCV to grab latest camera frame and store in self.current_frame

if not self.webcam.grab():
   raise ValueError('frame grab failed')
self.clear_buffer()

retval, most_recent_frame = self.webcam.retrieve(channel=0)
if not retval:
   raise ValueError('frame capture failed')
self.current_frame = most_recent_frame

We’re currently using OpenCV for photo capture in OS X as well, though it doesn’t seem to work as well as in Windows, so we’re looking for alternative tools we can use to capture photos from within the script.

Step Four. Face recognition with OpenCV

Once a photo is captured, it’s taken to OpenCV for face detection. A lot of things are happening in this method, so I’ve tried to annotate it as much as possible. draw_reticule is a helper method that draws targets of various styles.

def face_detect(self):
   # load image, then resize it to specified size
   img = self.current_frame
   img_w, img_h = map(int, self.opts.image_dimensions.split('x'))
   img = cv2.resize(img, (img_w, img_h))

   # initialize classifier with training set of faces
   face_filter = cv2.CascadeClassifier(self.opts.haar_file)

   # detect faces
   faces = face_filter.detectMultiScale(img, minNeighbors=4)

   # convert to grayscale then back, so that we can draw red targets over a grayscale
   # photo, for an especially ominous effect
   img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
   img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)

   # sort faces by size (we only use the biggest face for adjusting the camera)
   faces.sort(key=lambda face:face[2]*face[3])

   x_adj, y_adj = 0, 0
   if len(faces) > 0:
      face_detected = True

      # draw a rectangle around all faces except last face
      for (x,y,w,h) in faces[:-1]:
         draw_reticule(img, x, y, w, h, (0 , 0, 60), "box")

      # get last face, draw target, and calculate distance from center
      (x,y,w,h) = faces[-1]
      draw_reticule(img, x, y, w, h, (0 , 0, 170), "corners")
      x_adj =  ((x + w/2) - img_w/2) / float(img_w)
      y_adj = ((y + h/2) - img_h/2) / float(img_h)
   else:
      face_detected = False

   # output the modified image so that we can display it
   cv2.imwrite(self.opts.processed_img_file, img)

   return face_detected, x_adj, y_adj

Step Five. Displaying the target

After OpenCV has detected any faces, we display the modified image (converted to grayscale, with red targets drawn over faces).

This method has a great deal of platform-dependent code, to make it play equally nicely with Linux, OS X, and Windows:

In Linux, we open up ImageMagick display windows. These windows do not refresh automatically, so we kill any existing windows each time we open a new one.
In OS X, we open a Preview window. Conveniently, calling open -a Preview [path] refreshes the current Preview window.
In Windows, we open Windows Photo Viewer (this might not work in older versions of Windows). It refreshes itself automatically, so we only open a window the first time Camera.display is called.

# display the OpenCV-processed images
def display(self):
   if sys.platform == 'linux2':
      # Linux: display with ImageMagick
      if self.current_image_viewer:
         subprocess.call(['killall', self.current_image_viewer])
      subprocess.call("display " + self.opts.processed_img_file + ' &', shell=True)
      self.current_image_viewer = 'display'
   elif sys.platform == 'darwin':
      # OS X: display with Preview
      subprocess.call('open -a Preview ' + self.opts.processed_img_file, shell=True)
      self.current_image_viewer = 'Preview'
   else:
      # Windows: display with Windows Photo Viewer
      if not self.current_image_viewer:
         viewer = 'rundll32 "C:\Program Files\Windows Photo Viewer\PhotoViewer.dll" ImageView_Fullscreen'
         self.current_image_viewer = subprocess.Popen('%s %s\%s' % (viewer, os.getcwd(),
               self.opts.processed_img_file))

Step Six. Aiming and firing

If a face is detected, the turret adjusts itself to try to bring the face into the center of its field of view. The Camera.face_detect method returns x_adj and y_adj, which correspond to the distance from the center of the most prominent face (expressed as a fraction of the total width and height of the photo, respectively). These values are passed to the Turret.adjust method:

# adjusts the turret's position (units are fairly arbitary but work ok)
def adjust(self, right_dist, down_dist):
   right_seconds = right_dist * 0.64
   down_seconds = down_dist * 0.48

   if right_seconds > 0:
      self.launcher.turretRight()
      time.sleep(right_seconds)
      self.launcher.turretStop()
   elif right_seconds < 0:
      self.launcher.turretLeft()
      time.sleep(- right_seconds)
      self.launcher.turretStop()

   if down_seconds > 0:
      self.launcher.turretDown()
      time.sleep(down_seconds)
      self.launcher.turretStop()
   elif down_seconds < 0:
      self.launcher.turretUp()
      time.sleep(- down_seconds)
      self.launcher.turretStop()

Note that the only way to move the turret a certain distance is to estimate how long the turret’s rotation would last and send it commands to move and stop with the appropriate timing.

Finally, Turret.ready_aim_fire detects if the face was close enough to the center of the camera to fire, turning on the turret’s LED as a warning before firing a missile (if the --disarm flag is passed to the script, the LED is turned on, but no missile is fired). Then the loop continues, until the turret has fired all four of its missiles:

# turn on LED if face detected in range, and fire missiles if armed
def ready_aim_fire(self, x_adj, y_adj):
   if face_detected and abs(x_adj)<.05 and abs(y_adj)<.05:
      turret.launcher.ledOn()
      if self.opts.armed:
         turret.launcher.turretFire()
         time.sleep(3) # roughly how long it takes to fire
         self.missiles_remaining -= 1
         print 'Missile fired! Estimated ' + str(self.missiles_remaining) + ' missiles remaining.'
         if self.missiles_remaining < 1:
            raw_input("Ammunition depleted. Awaiting order to continue assault. [ENTER]")
            self.missiles_remaining = 4
      else:
         print 'Turret trained but not firing because of the --disarm directive.'
   else:
      turret.launcher.ledOff()

Try it yourself!

If you have a Dream Cheeky brand USB missile launcher (though it wouldn’t take much work to support other brands of missile launchers), a compact webcam, and a desire to build your very own defense system for your home or workplace, check out our GitHub repository.

We’ve gotten Sentinel working on Windows, OS X, and several Linux distros, though installing the dependencies (OpenCV, PyUSB, and others, depending on platform) can take some work.

We’re currently hard at work on some more features, including:

different modes of operation (such as a “sweep mode”, in which the turret continually pans until it locates a face, rather than staying idle when it doesn’t see a face)
a “kill-cam” feature that stores the pictures that it takes right as it shoots a target
easier installation of dependencies (especially on Windows)

Got any comments, questions, or suggestions? Be sure to let me know, either here or on Github.

Installing a Brother printer on 64-bit Ubuntu/Debian

2013-02-07T00:00:00+00:00

The other day I had to reinstall Linux Mint Debian Edition. The reinstall process went smoothly overall, with the only problem being, of all things, the driver for my Brother printer.

The Problem

The problem is that Brother’s official Linux printer drivers are all 32-bit, and don’t appear to play nicely with 64-bit Debian-based distributions.

When I tried installing the driver directly, I got an incompatible architecture error:

> sudo dpkg -i cupswrapperHL2270DW-2.0.4-2.i386.deb
dpkg: error processing cupswrapperHL2270DW-2.0.4-2.i386.deb (--install):
 package architecture (i386) does not match system (amd64)
Errors were encountered while processing:
 cupswrapperHL2270DW-2.0.4-2.i386.deb

Now, the recommended way to resolve this is to use MultiArch, Debian’s system for handling applications and libraries of multiple architectures on one system. However, MultiArch is still rather immature, and whenever I’ve tried to use it I’ve ended up in an unresolvable dependency hell. I wanted to find a better way to deal with this.

My first instinct was to suppress the incompatible architecture error using --force-architecture or --force-all:

> sudo dpkg -i --force-all cupswrapperHL2270DW-2.0.4-2.i386.deb
dpkg: warning: overriding problem because --force enabled:
 package architecture (i386) does not match system (amd64)
(Reading database ... 171933 files and directories currently installed.)
Preparing to replace cupswrapperhl2270dw 2.0.4-2 (using cupswrapperHL2270DW-2.0.4-2.i386.deb) ...
Restarting Common Unix Printing System: cupsd.
Unpacking replacement cupswrapperhl2270dw ...
dpkg: cupswrapperhl2270dw: dependency problems, but configuring anyway as you requested:
 cupswrapperhl2270dw depends on libc6 (>= 2.3.4-1); however:
  Package libc6:i386 is not installed.

Hmm, no luck yet - it looks like the driver depends on the libc6:i386 package, which I can’t install without MultiArch.

Interestingly, the Brother official FAQ suggests installing the lib32stdc++6 package (ia32-libs on Ubuntu) and using --force-architecture, but this still didn’t work for me, even after installing lib32stdc++6.

The Solution

Fortunately, a solution does exist, though it’s a bit hackish: you need to remove the libc dependency from the package.

There are a few ways to do this:

Option 1: Manual modification of the package

Yousry Abdallah recommends removing the libc dependency from the driver:

dpkg -x [package].deb common

dpkg --control [package].deb

vim DEBIAN/control

remove the "Dependency: libc (..." line (move to line than press 'dd' than ESC:x)

cp -a DEBIAN/ common/

dpkb -b common [package].deb

dpkg --force-all -i [package].deb

rm -rf common DEBIAN

Option 2: Official patch (Brother HL-2270DW only)

If you have a Brother HL-2270DW printer like me, then you’re in luck: Brother has released a patch to resolve the issue. You can follow the instructions here to use it.

Or you can simply download the already-patched CUPS driver, which has been patched by Chad Chenault. If you would rather use LPR, a patched LPR driver is also available.

Remember that you need to install the packages using the --force-all option:

> dpkg -i --force-all [package.deb]

I’ve tested the CUPS driver on Linux Mint Debian and it works perfectly. Both drivers should also work on Ubuntu 11.04 and later.

Finishing the installation

If you’re using the printer via USB, then everything should work after installing the package. Make sure your printer is plugged in, then go to http://localhost:631/printers and try printing a test page.

However, if you’re using a network printer, then there are still a few steps left. You can follow this tutorial or watch this video (skip to the part after the package is installed).

Good luck!

Quantifying Linguistic Diversity

2013-01-11T00:00:00+00:00

Adventures in Reinventing the Wheel

A few days ago, one of my friends posted this status on Facebook, and it got me thinking: how could I test a hypothesis like this? Supposing I were to use GDP per capita as a measure of affluence, I would simply have to compare it to a measure of linguistic diversity, and compute the correlation coefficient.

But does such a measure of linguistic diversity exist? I wasn't aware of one, so I decided to come up with my own, which I would call the Linguistic Diversity Index (LDI). I thought about a few potential ways to measure the diversity of languages in a country, and ulimately settled on this definition: the percentage probability that two random residents of a given country have different native languages. Compared to other potential measurements (such as the popularity of the most common language within a country), this definition for the LDI has the advantage of taking into account every language spoken in a country, while still being easy to visualize.

Computing the LDI is pretty straightforward. Given a country with population P and n languages, with L₁, L₂, ..., L_n speakers, respectively, the LDI of the country is:

Now, if I had bothered to do a Google search at this point, I would have found out that there actually is such a thing as a Linguistic Diversity Index (also known as Greenberg’s diversity index), and, moreover, it's defined more or less the same way that I defined it (namely, the probability that any two people of the country selected at random would have different mother tongues), and the data listed on the Wikipedia page is very close to what I ultimately obtained, as it was calculated using similar data. But I didn't realize that an LDI already existed at the time, and so I ended up essentially duplicating the existing work on the topic.

Scraping Ethnologue Data with Mechanize

Let me take some time to show how I collected the data for the LDI. If you'd rather just skip ahead to the pretty charts, go right ahead. I won't be mad.

Still here? Awesome.

In order to calculate the LDI, I needed to find the most accurate data possible on the number of speakers of different languages within each country, and so I turned to Ethnologue, a database of statistics on ~7000 languages. Each country's page in Ethnologue looks like this (with the red boxes indicating the numbers that I want to extract):

Doing this by hand would be incredibly tedious, so I needed to find a way to scrape all of the country pages on Ethnologue. I wasn't sure what the best way to do this in Ruby would be, so I asked my trusty assistant, howdoi:

> howdoi scrape web page in ruby
take a look at this library http://mechanize.rubyforge.org/

Ah, excellent! Mechanize does just what I want, and has spiffy syntax to boot (for example, languages = country_page / "#main table p" literally divides the page into an array of sections matched by the CSS selector #main table p). The full script that I used to gather all of the data I needed clocks in at around 50 lines, and the trickiest bit was coming up with the appropriate regexes to extract the right things:

require 'mechanize'

agent = Mechanize.new
language_regex = Regexp.new(/]\r\n\t\t\t\t\t\t([0-9,]*)(\.| )/)

index_page = agent.get('http://www.ethnologue.com/country_index.asp?place=all')

index_page.links_with(:href => /show_country.asp/).each do |country_link|
  country = country_link.text().strip()
  speakers = []

  country_page = agent.get('http://www.ethnologue.com/' + country_link.href)

  # get language speakers from table
  languages = country_page / "#main table p"
  if languages.length() > 0
    speakers += languages.map do |l|
      language_regex.match(l.children()[0].content())[1].gsub(',','').to_i rescue 0
    end
  end

  # get immigrant language speakers
  description = (country_page / "#main blockquote")[0].children()[0].content()
  immigrant_languages_description = description.scan(/Immigrant languages:.*?\./)[0]
  if immigrant_languages_description
    immigrant_languages = immigrant_languages_description.scan(/\(([0-9,]*)\)/)
                                                         .map{|x| x[0].gsub(',','').to_i}
    speakers += immigrant_languages
  end

  # get language speakers from subpages (if any)
  country_page.links_with(:href => /show_country.asp/).each do |subpage_link|
    subpage = agent.get('http://www.ethnologue.com/' + subpage_link.href)

    subpage_languages = subpage / "#main table p"
    if subpage_languages.length() > 0
      subpage_speakers = subpage_languages.map do |l|
        language_regex.match(l.children()[0].content())[1].gsub(',','').to_i rescue 0
      end
    end

    speakers += subpage_speakers
  end

  # compute diversity index
  total_speakers = speakers.inject(0, &:+)
  diversity_index = if total_speakers > 0
                      (1 - speakers.map{|x| x**2}.inject(0, &:+).to_f / total_speakers ** 2) * 100
                    else
                      "Insufficient data for meaningful answer"
                    end

  puts "#{country.ljust(40)}\t#{diversity_index}"
end

Note that I make a few simplifying assumptions here. For one thing, I didn't use Ethnologue's given value for the total population of each country, but rather set P = L₁ + L₂ + ... + L_n. This was important because I've noted Ethnologue's total population values to typically be more out of date than the language data, and I didn't want LDI values below zero, which could occur if, say, L₁ > P. Ethnologue language data is also imperfect - for example, some minor languages don't have their speaker count listed - but I assumed that the data was accurate enough as a whole for my computed LDI values to be meaningful.

Results

After running the script, I got these results. But raw data is boring. Let's make it into a map!

There's some interesting patterns that can be seen from the map. It appears that countries in Sub-Saharan Africa and South and Southeast Asia tend to have the highest rates of linguistic diversity (Papua New Guinea is at the very top, with over 600 different languages spoken and a diversity index of 99.03). All in all, thoughout the world, diversity appears to be the norm, rather than homogeneity: even Europe has its fair share of high-LDI countries (admittedly, a lot of this is made murky by the question of what is a language and what is a dialect).

Now I finally have all of the data that I need to test my friend's hypothesis about the relationship between language diversity and affluence. Using GDP (PPP) per capita data from the World Bank, I get the following plot:

There is a weak negative correlation (ρ = -0.247), suggesting that the relationship between language diversity and affluence is tenuous at best. Interestingly, while there are a great deal of countries that are both very poor and highly linguistically diverse, 7 of the 9 wealhiest countries (with the exceptions being Norway and the United States) all have LDI in the 48-78 range, perhaps suggesting that some level of linguistic diversity is economically useful in establishing a nation as a trade hub. Of course, correlation does not imply causation, and there aren't many conclusions that can reasonably be drawn from this chart without more outside information.

Conclusion

All told, this was a fun little project. It was unfortunate that I only found out later that the LDI already exists, and so my data collection was unnecessary, but it was nice to get some experience in scraping data, and I think that I've come up with some novel ways to present language diversity data. In the end, I feel that there is certainly room for more research to be done in the interconnection between the linguistic and economic spheres of countries.

Playing Around with Ruby Hashes

2012-07-30T00:00:00+00:00

As a relative Ruby newbie, one of the things I love about it is how easy and elegant it is to modify existing classes.

In this post I’d like to mention some of the cool things that I’ve discovered you can do with hashes.

Arbitrarily deep assignment

Suppose I’m using a hash to store data that has a complicated structure. I want to be able to do something like

hash[player][:stats][category][statistic] = value

without littering my code with a bunch of lines like

hash[player] ||= {:stats => {}}
hash[player][:stats][category] ||= {}

In other words, I want to be able to perform abitrarily deep assignment: for a hash h, h[:a][:b][:c] = 5 should be just as valid as h[:a] = 5, even if h does not yet have the key :a.

This is simple to implement by subclassing Hash and overriding the [] operator:

class FreeformHash < Hash
  def [](key)
    unless has_key?(key)
      self[key] = FreeformHash.new
    end
    super
  end
end

Let’s see it in action!

irb(main):009:0> h = FreeformHash.new
=> {}
irb(main):010:0> h[:a][:b][:c] = 5
=> 5
irb(main):011:0> h
=> {:a=>{:b=>{:c=>5}}}

One downside is that [] is now destructive, so h[key] will always create elements if they don’t already exist. Hash#has_key? should always be used where appropriate to avoid unwanted keys being created.

Default values

You might already know that Hash.new(X) will create a hash that returns X as a default value on a lookup for a nonexistent key:

irb(main):001:0> polite_hash = Hash.new("I'm so sorry")
=> {}
irb(main):002:0> polite_hash['not a key']
=> "I'm so sorry"

This is all well and good, but what if I want a default value that I can modify for each key? For instance, let’s say that I’m keeping track of some player statistics in a hash, and I want to be able to easily modify them without having to manually instantiate a hash for each player. My first thought is to specify a hash as a default value, as so:

irb(main):003:0> bad_hash = Hash.new({:wins=>0, :ties=>0, :losses=>0})
=> {}
irb(main):004:0> bad_hash[:player1]
=> {:count=>0}
irb(main):005:0> bad_hash[:player1][:wins] += 1
=> 1
irb(main):006:0> bad_hash[:player2]
=> {:wins=>1, :ties=>0, :losses=>0}

Uh oh, what happened here? It turns out that the default value of a hash is a single object that is used for all default values, so changing it once will change it for all default values, with nothing being stored in the hash itself:

irb(main):007:0> bad_hash.default
=> {:wins=>1, :ties=>0, :losses=>0}
irb(main):008:0> bad_hash
=> {}

Clearly this is not the way to go.

Fortunately, Ruby provides an alternative approach to default values: you can specify a block like {|hash, key| do_stuff} as an argument instead, and that block will execute every time a lookup fails. While no element is added to the hash by default in this case, you can use a block of the form {|hash, key| hash[key] = object} to insert key => object into the hash on every failed lookup.

A quick example:

irb(main):009:0> players = Hash.new{|h, k| h[k] = {wins: 0, ties: 0, losses: 0}}
=> {}
irb(main):010:0> players[:alex][:wins] += 1
=> 1
irb(main):011:0> players[:other_player][:losses] += 1
=> 1
irb(main):012:0> players
=> {:alex=>{:wins=>1, :ties=>0, :losses=>0}, :other_player=>{:wins=>0, :ties=>0, :losses=>1}}}

Deep sort

As of Ruby 1.9, hashes preserve their order of keys, but there are no methods for reordering keys. What do we do if we want to sort the elements in a hash by key?

Fortunately hashes include the Enumerable module, and thus have a sort method, but that returns an array of pairs, not a hash. To sort a hash by key and return a hash, we need to do:

Hash[h.sort]

where Hash[] constructs a hash with given input, such as from an array.

So far so good. Now, what if we have a hash like {y: {b: 2, a: 3}, x: {d: 4, e: 5, c: 6}} and we want to sort at each level, getting {x: {c: 6, d: 4, e: 5}, y: {a: 3, b: 2}} as a result?

This can be done in one line with a recursive function:

class Hash
  def deep_sort
    Hash[sort.map {|k, v| [k, v.is_a?(Hash) ? v.deep_sort : v]}]
  end
end

deep_sort works by first sorting the hash, then mapping over the elements of the hash, deep-sorting their values if possible, and finally converting the results back to a hash (because the mapping produces an array of pairs).

Now let’s see it in action:

irb(main):006:0> unsorted_monstrosity = {y: {b: 2, a: 3}, x: {d: 4, e: 5, c: 6}}
=> {:y=>{:b=>2, :a=>3}, :x=>{:d=>4, :e=>5, :c=>6}}
irb(main):007:0> unsorted_monstrosity.deep_sort
=> {:x=>{:c=>6, :d=>4, :e=>5}, :y=>{:a=>3, :b=>2}}

And there we have it: three simple tricks that serve as useful examples of just how elegantly powerful Ruby can be.

Dynamic Pluralization with PHP and jQuery

2012-07-30T00:00:00+00:00

(Note: I originally posted this on Facebook last summer, in my wild, PHP-loving days. While my toolbelt has changed since then, I still agree with the sentiment of the post, and so am reformatting it and reposting it here.)

Here’s a rather clean solution to dynamically pluralizing text that I’ve come up with. I wonder how other people deal with this.

Background

Web sites incorrectly using the plural form of nouns for dynamic content (e.g. “1 comments”) are a pet peeve of mine. Working on a site that displays lots of dynamic data that’s refreshed via AJAX calls made me want to come up with a clean way to deal with pluralization.

Of course, it’s easy to write a PHP function like:

<?php
public function pluralize($plural, $singular, $count) {
    return ($count = 1) ? $singular : $plural;
}
?>

and pass the current value of the data field in question. The necessity for client-side functions arises, however, when this data field dynamically changes (e.g. a user adds a comment and it’s automatically added to the page through AJAX), and then “1 comment” becomes “2 comment”. I couldn’t find a solution online that avoids especially messy JavaScript, so I came up with a couple of my own. Both approaches have the same general idea: a PHP pluralize() function is called to do initial pluralization and to create the HTML structure that can then be used by a JavaScript pluralize() function to refresh the pluralization whenever dynamic content is changed.

The Simpler Solution

This approach modifies the simple function above to create a pair of “singular” and “plural” spans and simply hide the one that’s not used at the moment.

PHP code for method #1:

<?php
public function pluralize($plural, $singular, $count, $id = "") {
    return ($count == 1) ?
        '<span class="singular'.$id.'">'.$singular.'</span>' . 
            '<span class="plural'.$id.' hidden">'.$plural.'</span>' :
        '<span class="singular'.$id.' hidden">'.$singular.'</span>' . 
            '<span class="plural'.$id.'">'.$plural.'</span>';
}
?>

JavaScript code for method #1:

function pluralize (value, id)
{
    if (typeof id == 'undefined') {
        var id = '';
    }

    if (value == 1) {
        $(".singular"+id).show();
        $(".plural"+id).hide();
    } else {
        $(".plural"+id).show();
        $(".singular"+id).hide();
    }
}

CSS for methods #1 and #2:

.hidden
{
    display: none;
}

To use this approach for a comment form like the one mentioned above, one would call pluralize("comments", "comment", $numComments) in the PHP, and then call pluralize (value) in JavaScript whenever the number of comments changes.

It’s a little messier to use this on a page with multiple dynamic fields (e.g. a reddit-style threaded discussion where every post can have “1 point” or >1 “points”), but still possible. The pluralize (value, id) function allows you to only modify the pluralization of one element at a time, as long as you pass a unique ID to the PHP pluralize() function calls.

This works, but it’s a little ugly to be left with a collection of spans with id’s like “plural1”, “plural2”, “plural3”, etc. For my second approach, I tried to get rid of this by structuring each “pluralization block” with a consistent hierarchy.

The More Involved Solution

The idea here is that the PHP pluralize() function takes a single parameter $contents of the form:

$contents = array(
    array('plain', 'There'),
    array('pluralize', 'are', 'is'),
    array('value', count($attendees)),
    array('pluralize', 'people', 'person'),
    array('plain', 'attending!')
);

and uses it to create an HTML hierarchy like this:

<span class="pluralizer-container">
    There
    <span class="singular hidden">is<span>
    <span class="plural">are</span>
    <span class="value">42</span>
    <span class="singular hidden">person</span>
    <span class="plural">people</span>
    attending!
</span>

The code for this is pretty straightforward:

<?php
public function pluralize ($contents) {
    $value = 0;
    $output = '<span class="pluralizer-container">';
    foreach ($contents as $element) {
        switch ($element[0]) {
            case "plain":
                $output .= $element[1] . ' ';
                break;
            case "pluralize":
                $output .= '<span class="singular">'.$element[2].'</span> ';
                $output .= '<span class="plural">'.$element[1].'</span> ';
                break;
            case "value":
                $value = $element[1];
                $output .= '<span class="value">'.$value.'</span> ';
                break;
        }
    }
    $output .= '</span>';

    if ($value == 1) {
        $output = str_replace('class="plural"', 'class="plural hidden"', $output);
    } else {
        $output = str_replace('class="singular"', 'class="singular hidden"', $output);
    }
    echo $output;
}
?>

The new JavaScript function takes advantage of this consistent layout to correctly pluralize all elements on the page at once without requiring any parameters:

function pluralize() {
    $(".pluralizer-container").each( function() {
        var value = $(this).children(".value").text();
        if (value == 1) {
            $(this).children(".singular").show();
            $(this).children(".plural").hide();
        } else {
            $(this).children(".singular").hide();
            $(this).children(".plural").show();
        }
    });
}

As a result, the JavaScript side of things is much easier to use, at the expense of a more complicated PHP call. This approach seems more elegant to me, but it does have some drawbacks: it takes some work to create the array that describes each “pluralization block”, and the JavaScript function acts on all pluralization blocks every time it’s called, which is more inefficient, though I imagine the performance difference is negligible.

To Wrap Up

Correctly pluralizing dynamic data is not especially difficult, and can be accomplished in some 40 lines of code. The two approaches I outlined both work fine and mainly differ in whether one wants to have slightly more complicated calls to the server function or the client function.

These approaches are easy to extend both to other languages (such as Russian, with its singular-dual-plural number system) and to other grammatical functions (such as present tense vs past tense for events). Another cool extension to this could be to integrate an automatic PHP pluralization function (I’ve seen a couple floating around online).

For those of you who use the Yii framework, I’ve encapsulated this functionality in a Yii widget for extra convenience. You can get it here: http://www.yiiframework.com/extension/pluralizer/

Refreshing Browsers over SSH

2012-07-25T00:00:00+00:00

Today I was faced with the challenge of having to refresh a Chrome browser window on a networked Ubuntu machine via SSH. (Admittedly, I could have walked over to the computer in question and hit F5, but where’s the fun in that?) This seemed like it would be simple, but I immediately ran into problems.

Opening Chrome

My first thought was to see if Google Chrome’s command-line interface has any support for interacting with existing windows, but unfortunately it doesn’t. Oh well - if the browser is set to remember the last opened tabs, then it should just be a matter of closing and reopening the window: once I SSH in,

sudo killall chrome
google-chrome & > /dev/null

should be all it takes, right?

Well, not quite:

(chromium-browser:3217): Gtk-WARNING **: cannot open display:

Hmm, Chrome refuses to start because it can’t find a display. Since we’re logged in remotely, we’ll need to manually set the display to be the one that is currently being used:

export DISPLAY=:0.0

After this, running google-chrome works. So far, so good.

Now comes the next problem: the original window was full screen. Could I make the new instance be full screen as well?

Making it full screen

Suprisingly, there is no command-line switch for full-screen mode! The closest I could find was –app, which didn’t seem to do anything. Well, if we can’t start the browser in full screen, can we at least make the window full screen after it starts?

Here the xdotool keyboard/mouse simulator package can come in handy - we can use it to mimic an F11 keypress as follows:

# You might need to sudo apt-get install xdotool
xdotool search chrome windowactivate --sync key F11

Now all that’s left is the matter of timing: we should only run xdotool once Chrome’s window has opened, so let’s give it a few seconds:

sleep 2

Putting it all together

The full sequence of commands to run is:

export DISPLAY=:0.0
sudo killall chrome
google-chrome & > /dev/null
sleep 2
xdotool search chrome windowactivate --sync key F11

Since I don’t want to have to enter this every time I need to refresh the browser, I saved this into a shell file on the remote machine at ~/local/bin/google-chrome-fullscreen and assigned appropriate permissions. Now, to refresh chrome on that machine, all I need to run is:

ssh <host> 'google-chrome-fullscreen > /dev/null'

(The > /dev/null is necessary, apparently because otherwise the SSH session hangs while waiting for output.)

What about other browsers?

Similar method should work for other browsers. A generic browser-refresh script that works for me is:

#!/bin/sh
# refresh_fullscreen

export DISPLAY=:0.0
sudo killall $1
$1 & > /dev/null
sleep 2
xdotool search $1 windowactivate --sync key F11

where the usage is

refresh_fullscreen <browser name or path>

I’ve tried this with Firefox and Chromium and it seems to work fine, but your mileage may vary.

About this blog

2012-07-13T00:00:00+00:00

It’s been a while since I’ve had a blog, but I thought that it’d be nice to have a place to post projects, ideas, tutorials, and whatever else is on my mind, programming-related or otherwise.

I’ve only ever used Blogger before, so this time around, I’ve decided to do things differently. I wanted a simple, self-hosted blogging system that would put me in control while still being mostly hassle-free. I think I’ve found a winner in Github’s own Jekyll. After starting with krisb’s Jekyll template and fiddling around for a few hours, I’ve found a whole host of things to love about Jekyll:

Why Use Jekyll?

It’s quick to install with RubyGems: simply gem install jekyll rdiscount compass and you’re done!
The domain-specific language for page logic is simple and Rubyesque, and templates can be nested as in Rails.
I get to write blog posts in Markdown, which is much more convenient than HTML for text-heavy content.
Thanks to the custom Rakefile included with the template, building and deploying is as simple as rake deploy, and since the generated site is static, it can be hosted absolutely anywhere, with zero configuration.

Admittedly, I’m not a big fan of the ERB-like layout files and would have preferred for them to be more like Slim or HAML, but since I imagine Jekyll is designed for rather simple sites, markup-wise, I don’t see this being too much of a problem.

Jekyll and Subdirectories

A more pressing issue is that Jekyll doesn’t seem to allow blogs to exist in subdirectories of a domain. I’ve experienced some issues trying to get this blog to run correctly on http://alex.nisnevich.com/blog, because the { { post.url } } values all treat http://alex.nisnevich.com as the root.

A temporary solution for me has been to replace all instances of { { post.url } } with /blog{ { post.url } }. This works remotely, but breaks the site locally, because running jekyll --server will still try to serve the blog at localhost:4000 rather than localhost:4000/blog .

I managed to circumvent this in the Rakefile via a rather hackish solution - an automatically generated symlink:

# (make_symlink is called from the build task)
@dir = 'blog'
def make_symlink
  sh 'cd _site && sudo ln -s . ' + @dir
end

Now _site/blog always points to _site locally, and all is well.

Well, not really. This is still an ugly hack, and I’d like to look into more long-term solutions - perhaps I should give Octopress a shot.

All in All

Aside from the subdirectory issue, Jekyll does succeed in minimizing unnecessary distractions and letting me focus on what I’m here for: the actual post content. Now all that’s left is for me to actually come up with that.

AlexNisnevich.blog

Reading List - 2025

★★★★½

★★★★

★★★½

★★★

★★½

★½

[unrated]

Reading List - 2024

★★★★★

★★★★½

★★★★

★★★½

★★★

★★½

★★

[unrated]

A Belated Wordbots Dev Diary / Retrospective (~1 Year Post-Release)

Table of contents

First, a visualization!

A (not so) brief history of Wordbots

The beginning (summer 2016–fall 2017)

Slow but steady improvement (fall 2017–summer 2019)

Distractions and demoralization (summer 2019–summer 2022)

The final push (fall 2022–spring 2023)

Aside: Wordbots as a Symbolic AI island in an LLM world

And finally, the release! (April 2023 and beyond)

What went right

1. Sticking through to completion

2. Investing in developer productivity

3. Soliciting feedback throughout

4. Me working throughout the whole stack, from the game client to the parser

5. The Wordbots concept itself

What went wrong

1. Struggling with motivation

2. Making a multiplayer game in the first place

3. Not thinking about the new player experience soon enough

4. Never finding a good way to split up work among a team

5. Spending too much time pursuing dead ends

6. Sticking to tools we knew

Final thoughts

Acknowledgements

Reading List - 2023

★★★★½

★★★★

★★★½

★★★

★★½

Buildling a DIY Tiki Window for a Home Tiki Bar

Step 1: The Screen

Step 2: The Housing

Step 3: The Bamboo

Step 4: The Rest of the Electronics

Step 5: Mounting

"Lingua o Dialetto?" Exploring Perceptions of Regional Italian Languages Over Time

Reading List - 2022

★★★★½

★★★★

★★★½

★★★

Extra! Extra! – «Never Been to a Party» Album Release

Reading List - 2021

★★★★½

★★★★

★★★½

★★½

★½

Reading List - 2020

★★★★★

★★★★½

★★★★

★★★½

★★★

★★½

★★

Migrating a ASP.NET Application from SQL Server to MySQL

Step 1. Copy over the Database

Step 2. Install New Packages

Step 3. Configure MySQL in web.config