Zack Hobson

Node.js and HTML5 Web Sockets

2010-03-28T00:00:00Z

Web Sockets elevate socket networking in client-side web apps to a first-class citizen, giving clients and servers a simple way to communicate over a persistent stream without the need for third-party plugins or hacks. Now that the developer release of Chrome supports Web Sockets, I decided to give it a try using node. Having never touched this technology before, it took me most of a day to get it all working. I am writing up some notes on the process for others who might want to try out something similar.

One major caveat: Node's networking code is in flux right now and which version you use can matter a great deal. For this exercise I'm using 0.1.33. It's worth noting that I had to downgrade from the edge version of node to get this working.

The Server

Besides node itself, there is a module called node.ws.js that wraps node's tcp socket API with a nearly identical interface: createServer() accepts a callback, and this callback receives an event listener for processing web socket events. Once a connection is made you can write to it like any socket. Messages from the client are processed asynchronously in the "data" listener.

var sys = require('sys'),
     ws = require('ws')

var server = ws.createServer(function (socket) {

  socket.addListener("connect", function (resource) {
    sys.puts("client connected from " + resource)
    socket.write("welcome\r\n")
  })

  socket.addListener("data", function (data) {
    socket.write(data)
  })

  socket.addListener("close", function () {
    sys.puts("client left")
  })
})

server.listen(8080)

Communication is initiated from the web browser with a special handshake sequence. This sequence looks like an HTTP header and is designed to allow an existing HTTP connection to upgrade to a web socket. We won't be taking advantage of this functionality, but there is a project called Socket.IO that provides some compelling features in this regard and may be worth a look.

The Client

Remember that in order for this to work your web browser must support web sockets, which is fairly rare at the moment. You will probably need to run a development version of Google Chrome.

Conceptually the client and server are nearly identical, as they're both reading and writing asynchronously to a network socket. In the client it's trivial to hook these callbacks up to the DOM using jQuery to do stuff like display server messages on the page.

I did quite a bit of digging before I stumbled on this great jQuery plugin called ws. The actual WebSocket API is quite easy to use, but I was going to be using jQuery in my client anyway, so why not. Certainly, the API for this plugin is pleasantly concise:

$(document).ready(function () {
  var ws = $.ws.conn({
    url : 'ws://localhost:8080',

    onopen : function () {
      console.log('connected');
    },

    onmessage : function (data) {
      console.log("received: " + data)
    },

    onclose : function (event) {
      console.log('disconnected');
    }
  });
});

This is pretty much the simplest possible thing you can do, but I hope this minimal example illustrates the potential of this technique. This works in the latest developer builds of Chromium or Google Chrome.

There are already some great examples of this kind of stuff in action. I recommend checking out this cool web socket demo from the developer of the ws jQuery plugin mentioned above. And of course there's WPilot, which ties web sockets together with the awesome HTML 5 canvas element.

HCl 0.3.0 supports free Harvest accounts!

2010-04-02T00:00:00Z

This morning I received two pull requests for the same issue in HCl, my command-line client for the Harvest time-tracking service:

They were both fixes for a long-running problem with free Harvest accounts that I just never got around to addressing. It surprised me to see this activity after so long, but when I checked my fork queue I found a number of commits for which I'd never received pull requests. What a nice surprise!

Anyway, thanks in part to the efforts of Michael Bleigh the latest version of HCl supports non-SSL (read: free) Harvest accounts. Yay!

Easier Sessions with MacVim

2011-02-21T00:00:00Z

Given: You never want to close your Vim session because you lose your lovingly crafted (haha!) buffer list, and in my case, open tabs. I am not here to convince you that you want to use sessions with Vim, that's up to you. Suffice to say that if you don't use editor sessions this post will not be of much use to you.

The Problem

I was thinking it'd be nice if it was a little easier to open Vim sessions on my Mac. Usually I use Alfred to start MacVim then "ESC-:session ~/Projects/blablah/Session.vim", tab-completing as appropriate. I can't just bind a key to this command because I have to specify the session file. If I am in a Terminal the process is pretty direct (I can use mvim -S filename) but usually I need an editor session before I need a Terminal.

Alternately, I can type "open session.vim" in Alfred to open the session file in a MacVim buffer and then "ESC-:so %", which maybe is technically easier, but I figured at that point I'm not far from being able to just open MacVim with that session automatically, right?

So I googled around a bit and found this useful article on the subject, which was almost exactly what I was looking for, with some caveats:

It requires that you save your session files with an extension besides ".vim" (I used the suggested ".vis").
The app kept popping up the "You haven't run this application before" dialog on every launch.
My bash environment is never loaded by the Automator app, so my path and such are not set appropriately. This messes with my Vim plugins that use external programs.

The Solution

Originally this post was about how to solve the problems listed above, which is possible to do in all three cases. However, I began to realize that there was probably a simpler way to accomplish this. This realization hinged on the fact that I didn't really ever want to edit my Vim session files, that this was what differentiated them from other Vim scripts.

So I simplified the process to the following steps (note that Automator is no longer involved):

Set up the Mac OS file associations such that .vis files are opened automatically with MacVim.
Add the following to your $HOME/.gvimrc:

" save sessions with .vis extension
map <leader>s :mksession!  session.vis<CR>

" automatically source vim sessions so I can open them with the finder
au BufRead *.vis so %

I decided to put this in my gvimrc since I don't really use sessions in console Vim (that's more for one-off stuff than project work), and it might be useful just in case I do need to edit a session file for some reason. Also I elected to stick to explicit session saving rather than autosave. Personally I don't mind saving sessions manually as I don't want to do it every time and I'm used to saving my files anyway.

So now, Alfred's open file functionality lets me instantly dial up any of my MacVim sessions:

When I select one of these it opens a MacVim window with the session loaded. A drastic reduction in keystrokes, and no need to involve Automator. Using a different file extension means that I can more easily select from all of my available sessions.

You don't need a plug-in to use ack with Vim

2011-09-02T00:00:00Z

ack is well known as one of the best command-line search tools, and there are plug-ins for many popular editors, including for some reason Vim. I've not been able to determine why these exist for Vim, as support is already built-in:

:set grepprg=ack\ -aH

See, Vim already has built-in support for external grep, so by telling ack to output the same format we can use it as a drop-in replacement:

:grep foo dir1 dir2

This populates the quickfix window, so you can use copen or cnext to list them or skip to the first one, respectively. In fact all the grep-related commands will work as usual, except that ack is doing all the work underneath.

Detached for Mac

2013-12-21T00:00:00Z

A version of this post originally appeared on the Cloud City blog.

Many developers are accustomed to working in a terminal window. Even on the Mac, with a high resolution display and accelerated graphics, much of the tools we use every day are operated through a terminal emulator. I personally worked this way before I used a Mac and I still do today. In fact, every Mac comes with a great tool for managing multiple terminal sessions: GNU Screen. Unfortunately, it can be hard to use, and it provides no visibility of your running sessions from the GUI.

Detached is a tool for your Mac that sits in your menu bar and lets you create terminal sessions that you can close and reopen later, without losing the associated process or your place in it (until you reboot). This is similar to how sessions work in most document-based applications: If you close the application with work in progress, you expect it to restore that state when you reopen it. Thanks to GNU Screen, this functionality is already built into your Mac: Detached just exposes it by making your terminal sessions accessible from your menu bar.

The first version of Detached was written about 3 years ago, using MacRuby. At the time, it was difficult to distribute MacRuby binaries, because they had to include the entire MacRuby runtime in order to be portable. The first distributable binary for the original version of Detached was over 70 megabytes!

It only took a few days to create the initial version of Detached, and once I'd gotten it chugging along it didn't need much work. It had been a fun experience, but I wasn't ready to build MacRuby apps for a living or anything. However, I still had Detached, and I ended up using it nearly every day for the next three years! By that time, I'd gone independent, there had been umpteen releases of Xcode, and the original Detached project was getting pretty hard to wrangle. I decided to rewrite it in Objective-C.

Rewriting was easy! Cocoa and Xcode have come a long way in the last three years and I had no trouble getting up to speed on ARC memory management and the like. The resulting binary was less than one megabyte, and two orders of magnitude smaller than the original. That's much easier to distribute!

The only thing I was missing was an icon for the application. I wasn't really keen on trying to do this myself, so I brought it to Stephanie Geerlings and Brendan Miller of Cloud City Development, who worked with me through several drafts before we settled on this fellow:

I love this logo because it's simple, memorable and charming. Earlier revisions tried to maintain some metaphor or theme relating to the software itself, but in the end it was more important to me that the image be appealing and fun. I think Brendan did a fantastic job.

My original plan was to distribute on the App Store, but I'd need to sandbox the app, which would mean not using AppleScript to power Terminal and iTerm, which is the core functionality! Instead, I am distributing Detached myself. The source code for Detached is also available under the terms of the GPL.

New features and fixes in HCl 0.4.x

2013-12-23T00:00:00Z

HCl has always just been about scratching an itch. I'd built it years ago when I needed to track my hours, and stopped maintaining it when I got a gig that didn't. In the late summer I started using Harvest again on a client project, and the experience made me realize that HCl still needed some work. The result was several weeks of bug-fixing and enhancements.

Improvements to task aliases

When I went back to using HCl, the biggest pain point was specifying and using task aliases. I had to look up the set command every time, and starting a timer felt cumbersome:

hcl set task.mytask 123 456
hcl start mytask some stuff

To address this, I made some changes to the start command, to make the command name optional and allow the task name to be specified anywhere with @. I also added a simpler alias command for aliasing tasks:

hcl alias mytask 123 456
hcl @mytask some stuff

Multiple Harvest accounts

You can use an alternate configuration by specifying HCL_DIR in the environment. This allows you to switch between multiple accounts quickly. For example, adding something like this to your shell configuration will create a myhcl command with separate account credentials:

alias myhcl="env HCL_DIR=~/.myhcl hcl"

You can then use myhcl and hcl with different account credentials to easily switch accounts.

Cancel a running timer

You can cancel a running timer (or the most recently stopped timer) with hcl cancel. You can also use hcl nvm and hcl oops.

Auto-completion of task aliases

You can enable completion of task aliases in Bash by adding this to your bashrc:

eval `hcl completion`

Refresh or review your Harvest credentials

You can now refresh your auth credentials with the --reauth option, and review them with hcl config. HCl also refreshes your credentials automatically on auth failure.

Log entries without a timer

At user request, there is a new command hcl log that you can use to log an entry without leaving a timer running:

hcl log @mytask +1.25 Finally did that thing.

MacOS X only: Passwords are now stored in your keychain

On MacOS X your password is now stored in your keychain! This behavior is automatic, you don't need to enable it. Unfortunately it's not available on any other platform yet.

More improvements and fixes since 0.3.x

Added support (and continuous integration) for several modern Ruby platforms.
The note command with no arguments displays all notes for a running timer.
Allow filtering of tasks by project code.
The stop command now checks for running timers from the previous day.
The resume command now accepts an optional task.
Added support for auto-retry on API throttle.

SSL and Ruby

2014-01-23T00:00:00Z

Update: Jeff Hodges has corrected a statement in this post about the most recent OpenSSL providing secure defaults in Ruby. The error was my fault, I misread part of the email thread in question. A follow-up post is in progress, but I've corrected this post in the meantime.

The ruby-core position is that they do not want Ruby to be responsible for the security of users. I and others believe that Ruby already is.
— Jeff Hodges (@jmhodges) January 17, 2014

Recently Jeff Hodges publicly disclosed the contents of a thread on the ruby-security mailing list. In this thread, Mr. Hodges pointed out that the lack of secure defaults means that users with older versions of OpenSSL will be susceptible to certain kinds of attacks. This generated some discussion in Ruby circles about the security responsibilities of Ruby's volunteer maintainers.

Yes, intentionally weakened rand # generators, suspect curves, & fiber taps are bad, but we're our own worse enemy: https://t.co/4y0tqeGyHz
— Kenn White (@kennwhite) January 21, 2014

At the same time, it's been pointed out by Kenn White that open-source Ruby software already has a pretty wide-spread issue with not validating the server SSL certificate against a root CA, allowing easy man-in-the-middle attacks. In this context, better defaults are not much help, as verification is explicitly by-passed.

Full disclosure: My Harvest command line had this bug before I switched to Faraday, which handles this behavior correctly by default.

It's probably important to you that your software uses SSL in the most effective way possible, so let's see how to avoid the above issues in your own Ruby programs and environment.

Using the latest OpenSSL version

Unfortunately, at the time of this writing, updating the version of OpenSSL used by Ruby will not necessarily provide the most secure defaults. Hopefully this will be fixed in Ruby itself.

To make sure your Ruby installation is using the latest OpenSSL, you need to know which version of OpenSSL your Ruby is using, and how to update it.

First, we'll invoke ruby on the command line to see which version of OpenSSL it was built with, and whether it matches the latest version listed on the OpenSSL web site:

ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'

Helpfully, this command reports both the version and release date of the OpenSSL library used by your Ruby installation. If it's out of date, you'll need to update OpenSSL and then rebuild Ruby.

The method for updating your OpenSSL installation and Ruby varies with your system. It's pretty straightforward on a Mac using Homebrew and rbenv:

brew update
brew upgrade openssl
rbenv install -f `rbenv version`

On my machine, this process ensured that my Ruby installation utilized the latest OpenSSL, which is 1.0.1f at the time of this writing.

Verifying your SSL certificate

In SSL, trust is every bit as important as secrecy. With no way to verify identity, you have no way to verify that someone else isn't just reading your data and passing it on. This is the reason we have root certs to begin with: They're the root of the trust network that verifies the owner of the domain is who they say they are.

In order to verify the SSL peer with Ruby's built-in OpenSSL and Net::HTTP, you need to specify the correct mode and cert storage location. Here is a simple example of a wrapper method that creates a Net::HTTP connection using peer verification:

require 'net/http'
require 'openssl'
def verified_peer_connect hostname
  Net::HTTP.new(hostname, 443).tap do |conn|
    conn.use_ssl = true
    conn.verify_mode = OpenSSL::SSL::VERIFY_PEER
    conn.cert_store = OpenSSL::X509::Store.new
    conn.cert_store.set_default_paths
  end
end

# Example: fetch the Google home page
conn = verified_peer_connect('www.google.com')
conn.get('/')

Of course, SSL will work without these settings, as those scores of open source examples can attest. The point is to more fully utilize the capabilities of SSL, including peer verification.

Ruby and Rails have been in the spotlight for security issues in the recent past, so it's a good idea these days to be especially vigilant about security updates and best practices. Regardless of any disagreement surrounding disclosure or default ciphers, these issues can be avoided entirely with just a little bit of up-front effort.

SSL and Ruby, part 2

2014-02-10T00:00:00Z

Recently I wrote about an issue with Ruby and SSL that was publicized by Jeff Hodges. Mr. Hodges has uncovered what seems like a fairly serious issue: Known insecure cipher suites and other options are being used by the OpenSSL bindings that ship with Ruby. In my original article I asserted that updating Ruby to the most recent version of OpenSSL will fix this issue, but this is not actually the case! There is still no officially published fix at the time of this writing, but there are ways you can fix this in your own Ruby installation, if you're so inclined.

Here is the leaked ruby-security thread in question, I still recommend reading it in its entirety. I'll attempt to summarize below. First a little background:

Cipher suites are named sets of security algorithms used to negotiate a secure connection. The SSL client maintains a list of these suites, to be attempted (in order) until the client and server can agree on a set of ciphers and establish a connection. Of course, weaknesses and vulnerabilities are found in encryption algorithms all the time. Better techniques are discovered. Best practices change. When this happens, it can render some cipher suites less than optimal for security. Ideally, the cipher suites using these algorithms would be deprecated, but servers are under pressure to support as many encryption techniques as they can, for the widest compatibility possible. This means that if the client prefers a known-insecure suite, the server is unlikely to refuse it.

Now then, the problem: Ruby specifies its own set of default cipher suites for OpenSSL, including some that have known problems. It also allows some other behaviors (like TLS compression) that are known to be insecure. In addition to this, the OpenSSL defaults themselves are not always secure. It follows from this that Ruby itself is the last line of defense for users of Ruby's OpenSSL bindings.

Some of Ruby's maintainers pushed back on this, arguing that Ruby is not a security project, and that bypassing OpenSSL defaults carries further risks. But as far as I can tell, this argument doesn't hold water, as Ruby already specifies its own default options and ciphers for OpenSSL. Someone proposed a patch that demonstrated which options and ciphers would be preferable, which gives us some insight into how we might solve this problem on our own.

If you'd like to see this problem for yourself, you can demonstrate the weakness on any Ruby installation using a simple Ruby program:

require "net/https"
require "uri"
require "json"

uri = URI.parse("https://www.howsmyssl.com/a/check")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
resp = JSON.parse(http.request(Net::HTTP::Get.new(uri.request_uri)).body)
puts JSON.pretty_generate(resp)

When I run the above code locally, I get a JSON-formatted report explaining the numerous problems with my SSL client, including a list of known-insecure cipher suites. As you can see, it's not looking so good (I've elided the complete list of cipher suites for brevity):

{
  "given_cipher_suites": [
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    <...>
    "TLS_EMPTY_RENEGOTIATION_INFO_SCSV"
  ],
  "ephemeral_keys_supported": true,
  "session_ticket_supported": true,
  "tls_compression_supported": true,
  "unknown_cipher_suite_supported": false,
  "beast_vuln": false,
  "able_to_detect_n_minus_one_splitting": false,
  "insecure_cipher_suites": {
    "TLS_DHE_DSS_WITH_DES_CBC_SHA": [
      "uses keys smaller than 128 bits in its encryption"
    ],
    "TLS_DHE_RSA_WITH_DES_CBC_SHA": [
      "uses keys smaller than 128 bits in its encryption"
    ],
    "TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA": [
      "is open to man-in-the-middle attacks because it does not authenticate the server"
    ],
    "TLS_ECDH_anon_WITH_AES_128_CBC_SHA": [
      "is open to man-in-the-middle attacks because it does not authenticate the server"
    ],
    "TLS_ECDH_anon_WITH_AES_256_CBC_SHA": [
      "is open to man-in-the-middle attacks because it does not authenticate the server"
    ],
    "TLS_ECDH_anon_WITH_RC4_128_SHA": [
      "is open to man-in-the-middle attacks because it does not authenticate the server"
    ],
    "TLS_RSA_WITH_DES_CBC_SHA": [
      "uses keys smaller than 128 bits in its encryption"
    ],
    "TLS_SRP_SHA_WITH_3DES_EDE_CBC_SHA": [
      "is open to man-in-the-middle attacks because it does not authenticate the server"
    ],
    "TLS_SRP_SHA_WITH_AES_128_CBC_SHA": [
      "is open to man-in-the-middle attacks because it does not authenticate the server"
    ],
    "TLS_SRP_SHA_WITH_AES_256_CBC_SHA": [
      "is open to man-in-the-middle attacks because it does not authenticate the server"
    ]
  },
  "tls_version": "TLS 1.2",
  "rating": "Bad"
}

Look at that, ten known insecure cipher suites! TLS compression! Rating "Bad"! There is good news, however: it's possible to provide your own list of ciphers to Net::HTTP when creating a connection:

require "net/https"
require "uri"
require "json"

uri = URI.parse("https://www.howsmyssl.com/a/check")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
# corrected ciphers
http.ciphers = "DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2"
resp = JSON.parse(http.request(Net::HTTP::Get.new(uri.request_uri)).body)
puts JSON.pretty_generate(resp)

If I run this code, I get a different report with no insecure ciphers listed, but unfortunately, the overall score for my TLS is still "Bad":

{
  "given_cipher_suites": [
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    <...>
    "TLS_EMPTY_RENEGOTIATION_INFO_SCSV"
  ],
  "ephemeral_keys_supported": true,
  "session_ticket_supported": true,
  "tls_compression_supported": true,
  "unknown_cipher_suite_supported": false,
  "beast_vuln": false,
  "able_to_detect_n_minus_one_splitting": false,
  "insecure_cipher_suites": {
  },
  "tls_version": "TLS 1.2",
  "rating": "Bad"
}

This is because there are other insecure options apart from the ciphers, and at the moment Ruby does not have hooks to set OpenSSL options from within Ruby code. There is a patch to allow setting options on the SSL context, applying it should allow us to provide our own, more secure options to SSL using Ruby. I've stolen and re-purposed a script that will install the patched Ruby using rbenv. If you'd like to try this out yourself, that script might be a good starting point.

Once you have a version of Ruby with the above patch applied, we can set SSL options in our TLS testing code, using the patch proposed in the original thread as a guide:

require "net/https"
require "uri"
require "json"

uri = URI.parse("https://www.howsmyssl.com/a/check")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
# avoid unsafe cipher suites
http.ciphers = "DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2"
# provide options for the SSL context
http.ssl_options = OpenSSL::SSL::OP_ALL | OpenSSL::SSL::OP_NO_COMPRESSION & ~OpenSSL::SSL::OP_DONT_INSERT_EMPTY_FRAGMENTS
resp = JSON.parse(http.request(Net::HTTP::Get.new(uri.request_uri)).body)
puts JSON.pretty_generate(resp)

When I run the above test code with a patched Ruby, the report improves:

{
  "given_cipher_suites": [
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    <...>
    "TLS_EMPTY_RENEGOTIATION_INFO_SCSV"
  ],
  "ephemeral_keys_supported": true,
  "session_ticket_supported": true,
  "tls_compression_supported": false,
  "unknown_cipher_suite_supported": false,
  "beast_vuln": false,
  "able_to_detect_n_minus_one_splitting": false,
  "insecure_cipher_suites": {
  },
  "tls_version": "TLS 1.2",
  "rating": "Probably Okay"
}

Probably okay! A positively glowing appraisal from our TLS testing service. This demonstrates what is possible with some fairly small changes to Ruby's OpenSSL bindings and the right SSL options.

While we're able to get some positive results in the end, this is still mostly bad news. Ruby's maintainers are pushing back on improving the known-insecure defaults already provided by Ruby's OpenSSL bindings, and there is no official release of Ruby that will even allow users to provide more secure options. On the bright side, the patch to add ssl_options to Net::HTTP is pretty non-controversial, and will probably make it into Ruby at some point. This at least will allow HTTP clients to work around it in their own Ruby code.

Become an internet DJ using pulseaudio

2020-05-26T00:00:00Z

This is a tutorial on how to use pulseaudio to stream the audio output of any program as an internet radio station.

I arrived at this solution because I have kind of a weird music set up: My collection of mostly Ogg Vorbis files are on a network-attached storage device in my apartment, exposed to the local network as a DAAP service. I can use rhythmbox to browse and play my music via this service.

When I became interested in internet radio, I learned that my setup wasn't conducive to streaming. Most tools that make broadcasting easy are designed to work with local files, but I specifically wanted to keep using rhythmbox and DAAP to play my music, even when broadcasting. It turns out that if you're already using pulseaudio, this is not that difficult to do! In fact, it comes with a significant upside: You can stream the sound from any program you want, and even include a microphone monitor!

Obviously there are a lot of ways to do this, but if you want to do things exactly as I did, this is what you'll need:

the radio station: an internet-accessible host for running icecast2, I use a Linode VPS
the recording studio: a computer that uses pulseaudio for sound, I use Ubuntu on a Thinkpad X1 Carbon
an audio player of your choice, I use rhythmbox but anything will do
the command-line tool darkice for streaming pulseaudio to icecast
the GUI tool pavucontrol (Pulse Audio Volume Control) for managing the inputs and outputs

The radio station

The first thing you'll need is an internet-accessible host for running your radio station. For this, I'm using a cheap Linode VPS with the name radio.butts.team pointed at it, on which I install icecast2 as root:

# as root on the icecast host
apt install icecast2

By default, icecast2 will start automatically and only needs to have the config updated to add the password we will use to authenticate when broadcasting:

# as root on the icecast host
vim /etc/icecast2/icecast.xml # edit authentication block to set passwords
service icecast2 restart

That's it! Your internet radio station is now waiting for you to start streaming. It's worth mentioning that at this point you can easily use a conventional player like Internet DJ Console to broadcast, and skip the rest of this tutorial.

If you want the flexibilty of using pulseaudio instead, what follows is a method for configuring your workstation to do just that.

The recording studio

NOTE The remainder of these examples are meant to be run on the machine you use to play music.

Step 1: broadcasting music

The trick to streaming via pulseaudio while still having normal use of your computer audio is to create a new sink specifically for your radio stream:

pactl load-module module-null-sink sink_name=Radio sink_properties="device.description='Radio'"

In pulse, sound comes from a source and outputs to a sink. This command creates a null sink named "Radio" that will be used to isolate the radio stream from other audio on the computer.

In order to stream that audio to your radio station for broadcasting, we'll use darkice:

sudo apt install darkice
cp /etc/share/darkice.cfg.gz .
gunzip darkice.cfg.gz
vim darkice.cfg

In darkice.cfg update the input section to use pulse:

[input]
device = pulse
# ...

Also, update the icecast2 config:

[icecast2-0]
server = <radio station hostname>
password = <radio station password>
# ...

The remaining config blocks in the file (for streaming to other types of service) can be deleted.

With this done, you can start darkice to enable streaming to your radio station:

darkice -c darkice.cfg

Now that we have an audio sink and a streaming source we can use pavucontrol to configure them:

sudo apt install pavucontrol
pavucontrol

In the "Recording" tab of pavucontrol (make sure "All Streams" are showing) you'll see a new device that streams all audio to your radio station. Set it to monitor the "Radio" sink that you created earlier.

Then, in the "Playback" tab of pavucontrol find the app that you want to stream from and set it to output to "Radio".

That's it! If everything worked, the audio from your seleted app is now streaming over the internet!

Step 2: the microphone monitor

At this point, we can use any app as an audio source for our internet radio station, but we don't yet have a way to use our microphone! To enable it, we need a way to use the microphone as an audio source that can be pointed at the Radio sink. This is accomplished by adding a loopback interface:

pactl load-module module-loopback latency_msec=1 sink=Radio sink_dont_move=true

Now in the "Recording" tab of pavucontrol there is another entry for the loopback interface, which you can then point at your microphone. You'll want to leave it muted except when you're broadcasting your mic, or your microphone audio will always be broadcast on the stream, probably not what you intended!

Step 3: make it permanent

Once all this is done, you might not want to re-do all of this the next time you restart your computer, so be sure to add your pulseaudio config to the end of /etc/pulse/default.pa:

# at the end of /ect/pulse/default.pa
load-module module-null-sink sink_name=Radio sink_properties="device.description='Radio'"
load-module module-loopback latency_msec=1 sink=Radio sink_dont_move=true

This will ensure that pulseaudio is configured on system start, but you will still need to start darkice and use pavucontrol to resume streaming.

Listening to the radio

Visiting your radio station host and port with a web browser will show you a useful interface for accessing your stream. Here's the icecast UI for my own radio station: radio.butts.team. I recommend using VLC, but any media player should be able to stream it.

Caveats

The default Ubuntu volume controls will mess up your stream if you touch them! If you don't stick with pavucontrol exclusively when you're streaming, a volume change will likely reset your music program to output to your speakers. This can probably be alleviated with the right pulseaudio settings but I haven't figure this out yet.

It's probably too easy to broadcast your mic accidentally, I might be able to alleviate this as well by setting it mute by defaut or something similar. I haven't figured this out yet either.

The last three years

2023-06-21T00:00:00Z

I was so excited to finally get to work at Workfrom! This was a company that Darren Buckner, Jewel Mlnarik and myself had co-founded a decade or so ago, but we learned eventually that there wasn't much need for a full-time technologist, so I went to work at New Relic.

While I'd spent a little time advising and consulting over those years, it wasn't until about three years ago that Workfrom realized that it was time to build a new product. This coincided with the end of my time at New Relic, so I finally got my chance: I went to be the head of product engineering at Workfrom.

By the time I arrived, Jewel had moved on and my counterpart in product was Brooke Hurford. With Darren and Jess Porec directing strategy and talking to existing community members, we started looking for new ways to enable collaboration for remote workers.

At first, I spent most of my time catching up on our existing deployed resources and reviewing code in the Places app that Darren and Jewel had built. I also kept a technical blog that I published nearly every day. It mostly contained research and info dumps about what I was working on, and why. I liked to joke during this time that I was basically a full-time blogger for two people.

Eventually we started prototyping some of our new ideas for connecting our community. We ran our own Jitsi Meet instance and experimented with embedding it in web pages. This was pretty fun, I specifically remember publishing my notes one day with a meeting embedded in it, so when Darren and Brooke went to read them I'd be there, just hanging out, in my notes! While this was a fun trick, it soon became clear that Jitsi Meet wasn't the best platform to build on. It wasn't designed to be customized in the ways we wanted, at least not without a deep knowledge of Jitsi's internals.

We were not interested in cobbling together or maintaining a real-time video and audio service by hand, so we looked around for other solutions. We soon found Daily, which offered a nearly perfect solution: Real time comms, embeddable in a web page, via a straightforward Javascript library dependency.

With Daily it was relatively easy to get going, and we had a working prototype pretty quickly. I cobbled together a stack based on Next.js, Blitz RPC, and React/Typescript for the application. DigitalOcean had just launched an "app platform" that we could hook up to GitLab for automatic deployment with Docker.

With the bones of the product taking shape, we started building out our dream hangout in earnest. Brooke and I cranked out one feature after another over two years as it matured into an actual SaaS product that we could offer to customers. During this time we had near-weekly production demos (we did our best never to demo unreleased features) and also the time of our lives. I love this work! Sometimes you just want clear requirements and a runway, and we had it. Customers loved it too! One user described it as "Zoom meets MySpace," which is a soundbite that I still use to this day.

This product has many happy customers now, you can get an idea of how it works by visiting CafeForTeams.com. So that's the last three years, more or less. What a great time it's been. I should republish some of those technical blog posts on here, probably.

A simple search component without external tooling

2023-08-31T00:00:00Z

I keep hearing that it's now possible to build self-contained web components using only native web technologies. Until now, creating a component that can be freely incorporated into another web page has required layers of inscrutable translators and compilers and bundlers between the code and the browser.

Here's the code in this page that enables the search interface above:

<notes-search
  data-src="/posts/index.json"
  data-placeholder="Search by title and press enter.">
</notes-search>

<script lang="javascript" src="/static/notes-search.js"></script>

It's just an element using a custom tag notes-search and a script tag to load the notes-search.js script that defines the custom element. Click the link above to take a look at the script. That's the entire thing, a web component in a single file that can be loaded on any web page!

The component

To build this component I used the custom element API and shadow DOM to create a little self-contained DOM that can be styled without affecting the rest of the page. It starts with a class definition:

class NotesSearch extends HTMLElement {

  constructor() {
    super();

    const wrapper = document.createElement('div');
    const form = wrapper.appendChild(document.createElement('form'));
    const input = form.appendChild(document.createElement('input'));
    input.setAttribute('name', 'q');
    input.setAttribute('autocomplete', 'off');
    input.setAttribute('placeholder', this.getAttribute('data-placeholder'));
    const button = form.appendChild(document.createElement('button'));
    button.textContent = 'Search';
    const results = wrapper.appendChild(document.createElement('ul'));

    // ...
  }

When the element is constructed, we create a wrapper div and populate it with the basic controls. The input placeholder is fetched from an attribute data-placeholder. This is my least favorite part of the code since I am used to using JSX or some other template language, but there isn't much of it.

We also set up the form submission and fetch the index data needed for the search:

    // Handle form submit.
    form.addEventListener(
      'submit',
      this._searchSubmitFormHandler(input, results),
    );

    // Select text on focus.
    input.addEventListener('focus', e => e.target.select());

    // We aren't using await because constructor isn't async.
    this._fetchNotesIndex()
      .then(index => {
        this.index = index;
      })
      .catch(e => {
        alert('Failed to fetch notes index.', e);
      });

This is not an advanced tool at all: For illustration purposes I generated a simple JSON list of my posts and I'm using that as the search index. A better component might have a search backend to call, but this is just a little static website.

Connecting it up

You may have noticed that the constructor didn't actually add anything to the document yet. For that, we are going to implement connectedCallback and utilize the shadow DOM API:

  connectedCallback() {
    this.attachShadow({ mode: 'open' });
    const style = document.createElement('style');
    style.textContent = this.style;
    this.shadowRoot.append(style, this.wrapper);
    this.input.focus();
  }

Here we call attachShadow, which defines this.shadowRoot, to which we append our style element and our wrapper element. Finally, we focus the search input.

Styling it

This value of this.style is a static string value on the class that contains all of the CSS for the component. Because we are only styling the shadow DOM, we don't have to worry about style conflicts or pollution, we can use whatever elements and class names we want!

  style = `
    :host {
      --form-border: #ccc;
      --form-bg: #fefefe;
      --focus-border: blue;
      --input-text: #000;
    }
    form {
      display: inline-flex;
      outline: 1px solid var(--form-border);
      border-radius: 0.25rem;
      padding: 0.5rem;
      width: 100%;
      background: var(--form-bg);
    }
    /* ... */
  `;

There are other ways we could load our styles. For instance, if you wanted your CSS to be in a different file (better syntax highlighting!), you could add a <link rel="stylesheet"> to your shadow DOM instead.

Also: The :host CSS selector is the shadow DOM equivalent of the :root selector, for the purpose that we are using it today. Here, I'm using it to define all the colors in one place.

Doing the thing

Our form submit handler is implemented in the method _searchSubmitFormHandler:

  // return a submit callback for the search form
  _searchSubmitFormHandler(input, results) {
    return async e => {
      e.preventDefault();
      if (!input.value) {
        results.replaceChildren();
        return;
      }

      const value = `${input.value}`.toLowerCase();
      const resultRows = this.index.notes
        .filter(note => note.title.toLowerCase().indexOf(value) >= 0)
        .map(note => {
          const row = document.createElement('li');
          const item = row.appendChild(document.createElement('a'));
          item.textContent = note.title;
          item.href = note.url;
          return row;
        });
      results.replaceChildren(...resultRows);
      input.select();
    };
  }

Given an input element and a results element, this method returns a handler that performs the search and displays the results. Again, the search itself is very primitive since it's mostly for illustration purposes. We also select the contents of the input element after the search, as a little treat.

Fetching the index

For completeness, here is _fetchNotesIndex, which is called in the constructor to load the index. It's just a wrapper around fetch that pulls the index URL from the data-src attribute of the custom element:

  // fetch the notes index
  async _fetchNotesIndex() {
    const notesUrl = this.getAttribute('data-src');
    try {
      const request = await fetch(notesUrl);
      return await request.json();
    } catch (e) {
      console.error('Failed to fetch notes index.', e);
      return {};
    }
  }

The Upsides

A component like this can be dropped into a legacy web app pretty easily, regardless of how it was made. I am going to try building more little components to get the hang of it. I didn't even get to incorporate <template> and <slot> yet.

The Downsides

I am already missing a couple of niceties from my usual React-based stack. The big one is that I can't use TypeScript, but I can see how it could be possible to incorporate with a relatively small increase in complexity (as compared to say, a full React app). I probably won't bother for small components like this, but I think it would make a difference if you wanted to use this technique in a large codebase.

I'm unsure how this tech could be used to construct an entire application, but it seems like it'd be possible to do. I like the idea of making an actual web app without all those translation layers.

Embedding Mastodon posts with a web native component

2023-09-03T00:00:00Z

This is the source code for the embed widget that (hopefully) appears at the top of this post. Here is how it's being used in this page:

<embed-post
  data-src="https://butts.team/@zenhob/110987984593500666"
  data-maxheight="325"
  data-maxwidth="800"
></embed-post>
<script async="async" src="/static/js/embed-post.js"></script>

As you can see, there isn't much to it. After my last post, where I learned about native web components, I started to wonder how hard it would be to use this technique to build a widget for embedding Mastodon posts. This is something I've never done before, but it wasn't that hard to figure out, so I wrote about the process. There are helpful links throughout, mostly to MDN, because that's my favorite reference. Since I already went into technical detail about web components in the last post, I will skip over many of those details here.

Figuring it out

First off, let's try adding .json to the end of the post URL and loading that in the browser. Sure enough, that displays a JSON representation of the post! Could it be that simple? Sadly, attempting to fetch that URL from within this page using fetch() fails with a CORS error. That makes sense: CORS is a safeguard to prevent web apps from making unauthorized requests on behalf of the user, and this interface isn't meant to be available to an outside web application. This seems like a dead end.

After an unsuccessful web search, I ended up poking around in the Mastodon code. Here we can find the code describing CORS behavior for Mastodon, and noticed that the /api path is excepted from CORS, indicating that it is for external use! It turns out there is an endpoint under this path that Mastodon provides explicitly for this purpose: /api/oembed. This is a pretty roundabout way to figure that out, but it didn't take long.

With this knowledge, we can fetch something related to the post, given the URL:

const hostname = new URL(postUrl).host
const resp = await fetch(`https://${hostname}/api/oembed?url=${postUrl}`)
const embed = await resp.json()

What we get is a response specifically for embedding, which contains a snippet of HTML. This snippet declares an <iframe> pointing at the original post, and loads some JavaScript. It looks something like this:

<iframe
  src="https://butts.team/@zenhob/110987984593500666/embed"
  class="mastodon-embed"
  allowfullscreen="allowfullscreen"
></iframe>
<script src="https://butts.team/embed.js" async="async"></script>

It's rarely a great idea to accept raw HTML and insert into your UI, there are serious security implications! For a service under my control, with my own post, this seems like something that can be done safely. Given that, we are just going to shovel it into the innerHTML of the component.

this.#content.innerHTML = embed.html

That works! At this point the post is visible through an awkwardly sized viewport on the page. The docs for /api/oembed explain how to specify the size of the embedded frame, so as long as our styles match those sizes, our embed should fit perfectly. The optional attributes data-maxwidth and data-maxheight are used to configure the styles and the API call.

Wrapping it up

By this point the bulk of core functionality exists, but there is still some polishing to do. User-provided maxheight and maxwidth values are applied using CSS variables in inline styles. I added a little "loading" message that's visible until the fetch completes and the embed is visible.

The code for this component makes use of some JavaScript features that the last one did not, like static and private members. This is mostly because I haven't really written much JavaScript lately and I am unused to our age of evergreen browsers, where I can use a more modern JS dialect without issue.

Type checking JavaScript with TypeScript

2023-09-05T00:00:00Z

Unlike the web component stuff, the thing I am curious about today is not especially new. Since I am accustomed to using TypeScript, I felt myself missing the type checker when working on straightforward browser-targeted JavaScript.

TypeScript for JavaScript

Just using TypeScript would not be a big leap here, I am already using a basic build system for the posting parts of this blog. However, I rather like that I can code directly to the browser! Also there are still ways to statically analyze JS code with the TypeScript compiler, so we're going to do that instead, just to see how useful it is.

Say you already have a web app with a couple JavaScript modules. The first step is probably to install TypeScript if it isn't installed yet, and to create a tsconfig.json:

$ yarn add typescript
$ cat > tsconfig.json
{
  "compilerOptions": {
    "target": "ES6",
    "allowJs": true,
    "checkJs": true,
    "noEmit": true
  },
  "include": ["static/js/*", "static/*.js"]
}

To get the tsc compiler to behave the way we need, we specify a few things. The target is ES6, which is the dialect of JS that we're using. allowJs and checkJs make sure that JS code is loaded and checked, otherwise it will only analyze TypeScript code. Finally the noEmit ensures that we aren't emitting any code, since we're just using the type checker. I am also putting my own paths in the include list.

Type-checking my code

Once I have tsc installed and configured I can run it with npx or as a command in my package.json:

$ npx tsc
static/notes-search.js:79:16 - error TS2339: Property 'select' does not exist
on type 'EventTarget'.

Oh hey, I got a type mismatch in my plain old JavaScript! It happens in an event listener, where I am selecting the search input text on focus:

const input = document.createElement('input')
// ...
input.addEventListener('focus', e => e.target.select())

The call to select() happens to be safe, because the event listener is being applied to an input element, which has that method. However, the type checker isn't buying it. There are a couple of ways to fix this!

We can cast to a known type using supported JSDoc annotations
We can ignore the callback args and capture input in the callback instead

First, let's try casting with JSDoc. This works because the TypeScript compiler utilizes JSDoc annotations in type analysis. Here, we're just casting the target object to the correct type before we use it:

input.addEventListener('focus', e => {
  const target = /** @type {HTMLInputElement} */ (e.target)
  target.select()
})

This works and is happily accepted by the TypeScript checker. It's a little noisy though. What about the second option?

input.addEventListener('focus', () => input.select())

This is accepted because the inferred type of input is already correct, so no cast is required. The type checker loves it! I like it too, it's even slightly clearer than the original code.

What about implicit `any`?

I'm already out of stuff to fix! But there is another knob we can mess with. By default, the TypeScript checker will implicitly apply the any type to values where no type is specified or inferred. I usually have the "implicit any" option disabled in my TypeScript code bases, let's see what happens when we do that here:

$ npx tsc
<...list of errors elided>

Found 12 errors in 2 files.

Look at that! There appear to be 12 places where the type of an expression is not able to be inferred. Of course, it's all member declarations and method parameters. Everything else is initialized with a value so it gets a type.

Most of these were actually fairly easy to add, until I got to some code that filters and maps over fetched data. I almost stopped here, but I became pretty curious about how I'd even do it. This is the code:

const resultRows = this.index.notes
  .filter(note => note.title.toLowerCase().indexOf(matchValue) >= 0)
  .map(note => {
    const row = document.createElement('li')
    const item = row.appendChild(document.createElement('a'))
    item.textContent = note.title
    item.href = note.url
    return row
  })

The problem here is that the note parameter for both inline callbacks cannot be inferred by the type checker. Since they both accept the same object type, I decide to declare it separately instead of repeating it:

/**
 * @typedef {object} Note
 * @property {string} title
 * @property {string} url
 */

const resultRows = this.index.notes
  .filter(
    /** @param {Note} note */ note =>
      note.title.toLowerCase().indexOf(matchValue) >= 0,
  )
  .map(
    /** @param {Note} note */ note => {
      const row = document.createElement('li')
      const item = row.appendChild(document.createElement('a'))
      item.textContent = note.title
      item.href = note.url
      return row
    },
  )

Not entirely sure this is worth it! This kind of thing would make it easier to use if it was a reusable module, but for a small self-contained component it feels like overkill to me. I do enjoy the warm embrace of a type checker, so I'll leave it in place.

It's nice to be able to use modern validation tools on my browser-targeted JavaScript code without committing to a bundler or build system. It shouldn't really be a surprise, since TypeScript and JavaScript are still semantically identical, but it's fun to see it in action.