What is Belkin Wemo?

Wemo was a brand of home automation devices produced by Belkin from 2011 through 2023. They released a few Matter/Thread devices after that and these are unaffected by this announcement.

The most-common Wemo device (in my experience) is their plug-in “Smart Plug” line, which was widely available at retail. I purchased four of their “Smart Plug Mini” devices at Costco a few years back and was very happy with their performance, functionality, and integration into Home Assistant. They also produced other smart devices, notably light switches, lights, and even crock pots, heaters, coffee makers, and baby monitors! Most of these devices required cloud connectivity for use and will become e-waste when Belkin cuts them off after January 2026.

Classic Wemo devices rely on Belkin’s cloud for configuration and everyday functionality, including integration with Amazon Alexa, Google Home, and IFTTT. All of this will cease to function following the Wemo cloud shutdown.

The complete list of Belkin devices, including their support and HomeKit status, is available in their Wemo FAQ.

Apple HomeKit to the Rescue?

Over time, Belkin added Apple HomeKit support as an option to many of their light switches, dimmers, and plugs, and this is independent of the native app and cloud connectivity. This support for Apple HomeKit offers a ray of hope: HomeKit devices connect locally over Wi-Fi with HomeKit Accessory Protocol (HAP) and this doesn’t require Belkin’s app or cloud!

Belkin’s Wemo FAQ is explicitly about this: “Wemo products configured for use with Apple HomeKit prior to January 31, 2026 will continue to function via HomeKit in the absence of Wemo cloud services and the Wemo app.”

This means you can “rescue” your compatible switches and plugs by enabling HomeKit support and continue to use them. But configuring HomeKit requires the Belkin Wemo app (at least for my Wemo Mini plugs) so you have to do this before the Wemo shutoff date. I recommend doing a factory reset and enabling HomeKit sooner rather than later to avoid losing your Wemo devices next year.

Enable HomeKit in Your Wemo Switches

Enabling HomeKit is a straightforward process as long as the Wemo app is still working. Regardless of whether or not the device includes the HomeKit logo and number on the back, the following devices appear to be compatible:

• WLS0403 – Wemo Smart Light Switch 3-Way
• WDS060 – Wemo Wi-Fi Smart Light Switch w/ Dimmer
• WLS040 – Wemo Smart Light Switch
• F7C064 – Wemo?HomeKit
• F7C059 – Wemo?Dimmer Light Switch
• F7C063 – Wemo?Mini Plugin Switch
• WSP090 – Wemo?Outdoor Plug
• WSP080 – Wemo Mini Smart Plug

Before you begin, make sure you have the Apple Home app installed and configured. I suggest doing a factory reset of the Wemo switch as well, just to make sure it’s really “clean” and ready to configure. Also, you might want to give it a few minutes after adding it to the Wemo app before trying to configure HomeKit.

1. In the Wemo app, tap the “… More” button at the bottom, then tap “Connect to Other Services”, then “Apple Home App”
2. Tap “Get Started” and the Wemo app will scan for compatible devices
3. Tap “Connect” next to a compatible Wemo device to start the process
4. This will bring up an Apple Home pop-up – select a Home (if you have more than one), then tap “Add to Home”, select a Location and tap “Continue”, give the switch a name and tap “Continue”, select Outlet and tap “Continue”
5. And you’re done!

The Wemo device will now show up in your Apple Home app and can be operated directly from there with no need for the Wemo app or cloud. It should continue to work after the cloud is shut down, but you will likely not be able to reconfigure the device after this date. At least you can avoid having to toss these Wemo devices into the e-waste pile for a while!

What About Home Assistant?

Home Assistant has a native Belkin Wemo integration which appears to operate locally without involvement of the Wemo cloud. This may continue to work without HomeKit after the shutdown date, since many people are already firewalling off their Wemo devices and using them exclusively from Home Assistant. We can’t be sure, but Home Assistant might be another way to keep your Wemo devices working past January.

You may also be able to access Wemo devices via HomeKit protocol by adding them as an Apple HomeKit Device in Home Assistant. If you aren’t an Apple ecosystem user, this might be a way to re-enable Wemo devices for your chosen ecosystem: Convert them to HomeKit, add them to Home Assistant, and present them to other devices. But this might prove too complex to be reliable or even desirable.

Stephen’s Stance

Proprietary products, especially cloud-based ones, pose real problems. This is especially troubling for devices that are physically installed like light switches or appliances. I have serious concerns about the long-term viability of connected devices and am turning away from anything that has a proprietary ecosystem, but it’s sometimes impossible to avoid them. Time and again, standards-based devices have longer useful lives than proprietary ones: The fact that Belkin’s Matter/Thread devices will continue to function, not to mention HomeKit compatibility, gives me some hope for the future of connected devices! I’m especially positive about open source devices using ESP Home, Tasmota, and so on, but will accept Matter devices as a minimum viable option. I encourage you to do the same!

© Stephen Foskett for Stephen Foskett, Pack Rat: Rescue Your Belkin Wemo with Apple HomeKit

Note: The “Jet” in JetKVM makes me think of the 1973 single by Paul McCartney & Wings, so I’m using their songs as headers in this article. Turn on some power pop to get the full experience.

Band on the Run

I’m a nerd and I have lots of computers. Web servers, routers, home automation, and a whole lot more. Too many, really. I built a server into an old Macintosh SE and built a giant file server. That sort of thing. I even repatriated all of my web servers (which I swear I will eventually document), my Activepieces instance, my YOURLS link shortener, and more. Kubernetes? Yeah, I got that in my coffee table.

The problem is, none of this is really production-ready. And it always seems to fall over whenever I go out of town. So I’ve been relying on a combination of Home Assistant-driven Z-Wave switches, big honkin’ batteries, jump hosts, and luck to keep things running. But sometimes all this can fail: What if the server is powered off? Or stuck at POST? And what if you need to adjust something in the BIOS screen? Wouldn’t it be amazing to be able to reset the power or plug in a USB drive from anywhere?

Sure some computers have out-of-band management built in: My FreeNAS TrueNAS server uses a Supermicro motherboard with (sketchy) IPMI, and I’ve been using a really nice HPE ProLiant server with built-in iLO. But most of my devices are sorely lacking when it comes to remote management! Think of JetKVM as iLO for Everyone!

Let ‘Em In

That’s where the JetKVM comes in. While most KVMs are focused on multiplexing your keyboard and monitor for local servers, the JetKVM is a new kind of device: It’s designed to allow you to access one server from anywhere. It’s a simple-looking device with USB, HDMI, Ethernet, and Extension ports on the back and a cute color screen on the front. The whole thing is packaged in a sturdy die cast housing that sits solidly in place. And it’s tiny – about the size of the power brick for your NUC or external hard drive.

I’ve had pretty good luck with Kickstarters (though there have been some busts…) and the JetKVM is no exception: It was delivered pretty much on schedule and worked perfectly out of the box. The software is surprisingly mature (though not perfect) and the whole experience has been a good one. I’ve got no qualm about giving one to a nerdy friend at Cloud Field Day!

Silly Love Songs

This is no 3D-printed beta-test monstrosity. Far from it: The JetKVM device is solid, textured, and feels better in the hand than 95% of IT gear out there. It reminds me of unboxing my first Drobo, with sleek smoked-glass solidity that inspires confidence. The packaging isn’t quite Apple-level, but I’m glad they didn’t waste time or money on that. It’s nice enough to be on store shelves and a heck of a lot nicer than the last KVM I bought!

Another nice aspect of the JetKVM experience is that it comes with everything you need to get started and plugs right in. There’s no need to read the manual or fiddle with configuration: Plug the mini-HDMI-to-HDMI cable and USB-C-to-A cable into your server, plug the Ethernet cable in, and you’re up and running. The little screen shows all you need to know to get started: The IP address and connectivity state are simply and elegantly displayed along with the MAC address and JetKVM logo. This is literally a 1-minute setup!

Coming Up

The software works great on mobile or desktop and thankfully seems solid in Safari. The device emulates a USB keyboard and mouse and can pass through your device keyboard or use an on-screen virtual keyboard. The HDMI video can emulate a few popular monitors if needed, and you can even upload your own custom EDID file if you need some specific resolution.

I do wish the EDID selection included a lower-resolution option, since the text can be hard to read on a browser window. I tend to pick the Dell D7271H since it’s 1080p, but I’ve been thinking of whipping up a 1024×768 option called “NEC MultiSync” for giggles!

No More Lonely Nights

The JetKVM also includes a handy virtual media function, allowing you to mount an ISO image uploaded to the device or from a URL. A forthcoming update will allow you to stream an image directly through your browser too!

This is incredibly useful for provisioning new servers. I set up a “crash cart” JetKVM device in the basement, connected it to a spare PC, and did a complete Ubuntu install from the comfy couch upstairs. It was much easier than writing the image to a USB drive and huddling over the machine.

Unfortunately, the virtual media function only emulates a USB optical drive, so I first had to reconfigure the BIOS to boot from CD/DVD. Happily this is pretty easy to do using the virtual Ctrl-Alt-Delete and function keys, which don’t require a two-finger combination like most modern “multimedia” keyboards. But I had to be quick about it, since it takes a moment for the HDMI to “sync” and show up in the browser, so the BIOS tends to fly by.

Live and Let Die

The JetKVM is so good as-is that I’m already dreaming of enhancements to this little platform. Here’s a smattering of wish-it-was-better ideas:

1. I’d love a full-screen mode, preferably with picture-in-picture on iPadOS
2. Power over Ethernet support would be incredible (it’s apparently in the works)
3. The whole power situation is a little odd: It has a power splitter cable, but when I power my “crash cart” using a USB-C wall wart the JetKVM reboots when the attached PC shuts down
4. The Ctrl-Alt-Delete button should be on the main screen since I often open the virtual keyboard just for this one key combo
5. I’ve had a few instances of the virtual optical drive getting “stuck” with media so it would be nice to be able to completely turn that on and off
6. Speaking of that, I’d love a reboot button since I’ve had the whole device lock up once
7. I discovered accidentally that the cute little screen is a touchscreen, though the swipe-in-from-left menu only gives moderately-useful device info
8. I wish it had a full-sized HDMI port instead of the maddening mini-HDMI; at least it’s not a micro!
9. I hate that JetKVM Cloud can only authenticate using a Google account

One more thing: That RJ11 Extension port is very cool, promising incredible control over an ATX PC (using the ATX Power Control board). But I’m still waiting on my ATX extension boards so I couldn’t test it out. In theory the board sits between the power supply and motherboard, enabling you to remotely hard- and soft-reset the PC. And it powers the JetKVM even when the PC is off! But this is all theoretical since my boards are nowhere to be seen and apparently the software isn’t ready anyway. I’ll update this post when/if they arrive!

Listen to What the Man Said

The JetKVM is an excellent little device, and a real steal at $69. It’s wonderfully useful right out of the box and the planned software updates ought to make it even better. If you’re like me and have a few critical PCs without integrated out-of-band management (like HPE iLO or Supermicro IPMI) this little device is a lifesaver. And it even offers a few tricks those expensive enterprise solutions don’t offer!

© Stephen Foskett for Stephen Foskett, Pack Rat: JetKVM Gives You a Keyboard, Video, and More From Anywhere

Monitoring Linux Server Stats

I’ve been a Unix admin for over 3 decades, and have spent much of that time fighting with various software packages to monitor server stats. SNMP is anything but simple, and commercial packages traditionally made a significant dent on CPU and memory resources. Then there was the issue of collecting, formatting, and filtering stats to create a useful dashboard.

Home Assistant excels at collecting information, organizing it in a database, and displaying useful graphs and dashboards. But it’s not meant to handle system statistics – Home Assistant is primarily a platform for IoT devices. Although there is an official System Monitor service, it’s intended to monitor the local server running Home Assistant, not a remote server.

Happily, there is an excellent lightweight system monitor package designed for exactly this purpose: Linux2MQTT collects system stats and exports them via MQTT. As developer Cyrill Raccaud says, “linux2mqtt is a lightweight wrapper around psutil that publishes CPU utilization, free memory, and other system-level stats to a MQTT broker. The primary use case is to collect system performance metrics for ingestion into Home Assistant (HA) for alerting, reporting, and firing off any number of automations.” I’m happy to report that it works quite well, even if it is a little fiddly.

In case you’re not familiar with MQTT, it’s a lightweight publish-subscribe message queueing protocol that’s found a niche in the IoT space. MQTT clients pass messages through an MQTT broker using “topics” that other clients can subscribe to. Home Assistant has an open source MQTT broker Add-On called Mosquitto as well as an MQTT client, and many IoT devices like my Tasmota lights and Shelly switches already use this protocol for control and metrics. A client like linux2mqtt can easily publish system stats to a broker like Mosquitto for use in Home Assistant.

Installing and Running linux2mqtt in Ubuntu

The linux2mqtt package is written in Python and is available on Pypi, making it easy to install and maintain. But I do have a few tips and tricks to share to get it up and running monitoring Ubuntu servers.

A best practice in Debian and Ubuntu (and frankly all Unix systems) is to install Python packages in a virtual environment to maintain proper version of the Python interpreter and libraries. This functions something like a container (though less isolated or portable) and overcomes many of the headaches of maintaining a usable Python environment.

Note that linux2mqtt runs as a user, not as root. This should be reassuring but also poses some issues we will overcome!

Creating a venv for linux2mqtt is our first step:

sudo apt update
sudo apt install python3-venv
python3 -m venv ~/linux2mqtt

Now that we have a proper venv set up in our user home directory we can install linux2mqtt inside:

~/linux2mqtt/bin/pip install linux2mqtt

We now have a version of linux2mqtt installed and ready to test out!

Preparing Home Assistant

Configuring Mosquitto and the MQTT client in Home Assistant is far beyond the subject of this article. Suffice to say, the built-in Home Assistant Mosquitto broker and client are sufficient for everything we are doing here, though you could also use any other broker if you choose.

The important thing is that you have the IP address, username, and password of your MQTT broker. For purposes of this illustration I will use the following:

• MQTT Broker IP: 192.168.1.31
• MQTT Broker User: mosquitto
• MQTT Broker Password: password

Make sure the MQTT client is up and running in “Devices & Services” and is configured to query the Mosquitto broker. By default, Home Assistant uses “homeassistant” as the MQTT auto-discovery prefix, and this is also the default for linux2mqtt. If you change this prefix you will need to specify the new one using the –homeassistant-prefix command line parameter.

Happily, Home Assistant will properly receive and organize incoming data from linux2mqtt without any other setup. As soon as you start publishing system data to the broker a new device and associated entities will appear in Home Assistant!

Selecting System Metrics

It’s a good idea to pick out the specific system metrics to send to Home Assistant before running linux2mqtt for the first time. For the purposes of this article, we will send CPU percentage, CPU temperature, network utilization, and filesystem usage. Some of these requires a little preparation work, so let’s dive in!

We’ll build a metric set to send to the MQTT broker interactively, but here’s a good baseline:

~/linux2mqtt/bin/linux2mqtt \
--name $HOSTNAME \
--cpu=15 \
--vm \
--temp \
--fan \
--du='/' \
--du='/home' \
--net=enp0s1,15 \
--host=192.168.1.31 \
--username=mosquitto \
--password=password \
-vvvvv

One of the most important command line parameters is “–name $HOSTNAME“. This identifies all of the state metrics in MQTT and Home Assistant. You can use $HOSTNAME (as I did here) or specify a name manually. I like Initial Capital Letters so I generally specify the hostname manually.

The easiest sensor to add is CPU. Just add “–cpu=60” to the linux2mqtt command line and it will send overall CPU usage plus a number of detailed state attributes (see below) each minute. I actually like to use “–cpu=15” to get more frequent updates (every 15 seconds) but you’re free to choose any value you like.

Another useful sensor is virtual memory. Add “–vm” to the command line to send virtual memory usage plus detailed state attributes. I wish there was more instrumentation for physical memory, but I can’t find it.

Each server platform will have its own set of temperature metrics, and mine are all over the map. Some provide a ton of detailed thermal info with “–temp” for various zones while others just show the CPU. Not all systems will show fan status, but it doesn’t break anything to add “–fan” just in case. If your server doesn’t have fan sensors you could leave this off the final parameter list.

Next we want to collect disk usage stats. Most people will add “–du=’/’” to collect root directory stats. Being a veteran sysadmin and storage nerd I like to use separate filesystems for things that might fill up, including /home, /var/lib/docker, and various application directories in /srv. Just run “df -h” on the command line and decide which volumes you want to include. Then add multiple parameters like “–du=’/home’” to the list.

You can also monitor one or more network interfaces. You will need to specify the specific interface name on the command line, and the easiest way to get this is by typing “ip address show | more” on the command line. If you’re using Docker or similar tools you might have quite a few! I’ve simplified it above to just “–net=enp0s1,15“, which will send stats for the interface called enp0s1 every 15 seconds. You can add multiple parameters with different network names to monitor multiple interfaces.

Next we have to specify the MQTT Broker with “–host=192.168.1.31” and authenticate with “–username=mosquitto” and “–password=password“. Obviously you’ll use your own broker IP address, username, and password.

Finally we can add “-vvvvv” to show verbose debugging info on our first run. We will leave this out when we put this into production.

Now that you have figured out which parameters to use for your particular host, you can simply run linux2mqtt on the command line:

~/linux2mqtt/bin/linux2mqtt --name $HOSTNAME --cpu=15 --vm --temp --fan --du='/' --du='/home' --net=enp0s1,15 --host=192.168.1.31 --username=mosquitto --password=password -vvvvv

This will start collection and begin sending data to the MQTT broker. Within a few seconds you will see this show up in Home Assistant under the MQTT client! Once you’re satisfied you can stop this initial run with ctrl-c and proceed to put linux2mqtt into production!

Creating a Systemd Service for linux2mqtt

I am not a fan of systemd but that’s what Debian and Ubuntu use to run system services. And it works. So we’re going to use it.

Although many of us have mucked about with system services, I was less familiar with using systemd to run user-space services. But that’s the right way to run linux2mqtt, since it does not need root privileges.

We’ll create a systemd service file in our home directory for linux2mqtt:

mkdir -p ~/.config/systemd/user
vi ~/.config/systemd/user/linux2mqtt.service

This service file will be pretty simple. The most important bit is the command used to run linux2mqtt, which we developed in the previous section. Copy whatever worked at the end there as the “ExecStart” command in the following template. You should specify the complete directory path rather than “/home/username” and the proper hostname in this file. I’ve bolded these below. And don’t include the “-vvvvv”.

[Unit]
Description=Log system information via MQTT
DefaultDependencies=no

[Service]
ExecStart=/home/username/linux2mqtt/bin/linux2mqtt --name Hostname --cpu=15 --vm --temp --fan --du='/' --du='/home' --net=enp0s1,15 --host=192.168.1.31 --username=mosquitto --password=password
Type=exec
Restart=always

[Install]
WantedBy=default.target

Assuming you’re ready to roll, just enable these using systemd and you’re done on the host side! Note that you need to enable “linger” so systemd can run things when you’re not actively logged in.

sudo loginctl enable-linger $USER 
systemctl --user daemon-reload
systemctl --user enable linux2mqtt.service
systemctl --user start linux2mqtt.service

Home Assistant State Attributes

The first thing to understand about linux2mqtt is that it sends a basic state plus a number of state attributes for each sensor. For example, the CPU sensor displays percentage used by default but also sends User, Nice, System, Idle, and so on as “Attributes”. These can be seen in the Home Assistant UI by clicking the Attributes dropdown box below the sensor History graph. You can show these attributes in some (but far too few) dashboard cards, or you can create a helper template to use them anywhere.

I don’t love this. But it’s what we have to work with.

The easiest thing to do is simply use the basic/default sensor attributes when building Home Assistant cards. This works pretty well for CPU and temperature sensors, and I’m using those as-is. Total Rate isn’t too bad for network usage, but I would prefer to show transfer and receive rate. And I’m really not happy with a display of bytes used as my disk statistic.

Happily, the default Entity Card can show Attributes right from the GUI. I just selected Percent under Attributes, added “%” as the Unit, and added a sensible Name. Once you add this, Home Assistant will even show you a proper historic graph for this Attribute when you click on the value. Nice!

Looking in the Code Editor, we can see how the Entity Card specifies this Attribute in yaml:

type: entity
entity: sensor.pet_linux_disk_usage_volume
attribute: percent
unit: "%"
name: /

This gives us clues as to how to present this information in other Cards. Some can handle Attributes if you enter them in the Code Editor, while others simply can’t display them. Sadly the Glance Card does not support Attributes, but I was able to create some slick Cards!

The linux2mqtt page suggests using kalkih’s mini-graph-card to display graphs of sensor data, and I found it to be quite attractive once I customized it. This Card is available in HACS – just search for mini-graph-card!

Although mini-graph-card doesn’t support visual editing, it’s fairly easy to add Attributes to its graphs. For example, here’s my yaml for a slick network graph:

type: custom:mini-graph-card
entities:
  - entity: sensor.pet_linux_network_throughput_nic_enp2s0
    show_graph: false
  - entity: sensor.pet_linux_network_throughput_nic_enp2s0
    attribute: tx_rate
    name: Transmit
  - entity: sensor.pet_linux_network_throughput_nic_enp2s0
    attribute: rx_rate
    name: Receive
hours_to_show: 24
decimals: 0
name: Pet Network
lower_bound: 0
smoothing: false
show:
  legend: false
  fill: false
  points: false

The resulting card shows aggregate throughput as well as transmit and receive graphs for the host Pet. You’ll note that the basic aggregate throughput is displayed but not graphed while the transmit and receive stats are graphed on the same scale. I hid the legend, points, and fill since they were simply visual clutter.

Here’s a nice mini-graph-card setup for CPU and memory. Substitute your own sensor IDs as needed:

type: custom:mini-graph-card
entities:
  - entity: sensor.pet_linux_cpu
    show_graph: false
  - entity: sensor.pet_linux_thermal_zone_k10temp_tctl
    show_state: true
    show_graph: false
  - entity: sensor.pet_linux_virtual_memory
    attribute: percent
    unit: "% RAM"
    show_state: true
    show_graph: false
  - entity: sensor.pet_linux_cpu
    attribute: user
    name: User
  - entity: sensor.pet_linux_cpu
    attribute: system
    name: System
  - entity: sensor.pet_linux_cpu
    attribute: iowait
    name: Wait
hours_to_show: 24
decimals: 1
name: Pet CPU
smoothing: false
show:
  fill: false
  points: false

Stephen’s Stance

This little project of adding remote system stats to a Dashboard illustrates what I love and hate about Home Assistant. It’s an excellent platform for collecting and displaying metrics and has an incredible ecosystem of add-ons and supported protocols. But actually getting what you want can be frustrating and fidgety, as witnessed by the spotty support for Attributes in various cards. It’s frustrating that Attributes are not supported in the Glance card or the new Badges, for instance. But I was ultimately able to make it all work, and the result was worth the effort.

© Stephen Foskett for Stephen Foskett, Pack Rat: How to Monitor Linux Server Stats in Home Assistant

I’ll begin with an overview of what’s available for home and small office power backup. This is by no means exhaustive, but it reflects the realistic options I found when researching. I welcome your suggestions in the comments!

Office UPS Units

By far the easiest approach to dealing with power outages is to use a consumer un-interruptable power system (UPS). These are widely available and start around $100, making them easy to deploy.

The biggest issue with consumer UPSes is that they lack any real long-term power capability. Despite the big numbers on the box the run time for a UPS is measured in minutes rather than hours, making them an incomplete solution if the power is out for long. Even larger or more expensive ones only last a few hours, and running them down to zero will destroy the expensive batteries inside. This is the other big issue. Expect to spend about the purchase price every few years to replace the sealed lead-acid batteries. Most UPS controls can’t actually detect a bad battery so you’re often surprised when your UPS instantly dies without warning. UPS units also can’t power high-wattage devices like the sump pump that keeps my basement dry. And their AC inverters are cheap, noisy square wave “choppers” so some equipment might not like the power.

Still, UPS units are a great way to quickly cover short-term power outages and can make a nice complement to a bigger power solution. In fact, I spent some money replacing the batteries in a few UPS units to make sure the rest of my solution is truly uninterrupted!

Gas-Powered Generators

One popular solution to longer-term power outages is a generator. Smaller portable units typically cost around $1000 while large fixed generators can cost more than 10 times that much. They usually use liquid fuel (gasoline) or gas (propane or natural gas) to power a small piston engine (air or liquid cooled) and run an AC inverter. Portable generators usually have a few conventional AC outlets, so you would run extension cords inside to keep a few key appliances running, while larger units are usually hard-wired to the house.

Because they burn fossil fuels, generators must be vented and usually are placed outdoors when running. This complicates installation and use. Portable generators are usually stored in a garage or shed and rolled out manually when needed. So it takes a good hour to get up and running, since the generator must be moved into place and fueled and extension cables must be run to necessary appliances. As anyone with a lawnmower knows, small engines like these require maintenance and gasoline (even with fuel stabilizers) can go bad over time. Unless you are careful to regularly test and maintain your generator you might find that it won’t start or run reliably when you need it most! Propane is much better in this regard, since it is more stable and portable. Natural gas hookups are even better. But the engines still require maintenance (oil changes, air filter cleaning, etc) regardless of fuel.

“Whole house” generators are permanently installed, typically on a slab of concrete, and wired to a transfer switch inside the house. I talked to many friends and neighbors who spent the money (typically $10,000 to $20,000) for a complete whole-home generator setup and none was happy with theirs. The best setups use an automatic transfer switch (see below) and power on once a month, yet they often failed to perform as expected in outages. I’m sure people will disagree in the comments (especially dealers of whole-home generators) but the reviews of these solutions are awful. A family member is on their third unit and recently found that it needs to be replaced again! On the plus side, whole-home generators can power the entire house and (in theory) can be activated in a few minutes in the event of an outage.

One common issue with generator-based solutions is fuel consumption. Although manufacturers no longer publish “idle” numbers, it is clear that they are most efficient under moderate to heavy load. Most will “drink” 25% or even 50% as much at light load as at full throttle. This might not be a problem (apart from cost and pollution) when connected to a natural gas pipe or large tank, but it’s a real concern for portable units with small tanks. Because it’s not a good idea to store gasoline for months or years, be prepared to run to the station to fill up when the power goes out! This is one reason to consider propane instead: It’s stable for years and a 20 lb tank runs as long as 5 gallons of gasoline.

Solar Panels

Solar panels are amazing, but they don’t solve the problem alone. One of the common misconceptions about solar panels is that they will power your home or business in the event of an outage. In fact, this is illegal in most locations and is simply not practical. Solar panels are an excellent power source, help reduce greenhouse and CO2 emissions, and cost less than you would think. But most installations are grid-connected and do not provide power in the event of an outage, even on a sunny day! This would require the addition of a battery and transfer switch, which is much more costly and complex, as we will discuss.

Solar panels have really improved over the last few decades, and today’s panels can provide an incredibly amount of electric power at very low cost. In fact, it is now cheaper to buy and install solar than to use conventional power for just a few years. But the power they provide varies dramatically moment-to-moment. A shadow on a panel can reduce or stop production, and they must be properly angled to maximize output. And of course they only produce electricity in direct sunlight. Therefore most modern solar deployments are connected to the electric grid, offsetting your demand or even powering the local grid when the sun is shining. Many also use “micro inverters” to output AC from each panel to reduce the impact of local shadows or under-performing panels on the rest of the system.

For the purposes of this article, I will lump solar panels together with the public grid as a power source but not as a solution to cover mid- to long-term power outages for the home or office. I strongly recommend investing in solar, however, since it brings many benefits and doesn’t cost as much as you think! Many in the DiY community are buying bulk or surplus panels for well under $1000 and connecting them to the expensive systems listed here.

Larger Battery Systems

I’ve mentioned batteries a few times so far, so it’s time to tackle the peculiarities of these. When considering battery-backed power, it’s important to consider the cost and benefits of the various options. There are small battery systems like the UPS mentioned earlier and larger solutions like the Tesla Power Wall and similar devices. Then we must consider chemistry, from lead-acid to lithium-ion, and the variations on each.

Since we covered consumer UPS devices above, let’s focus only on larger units capable of supplying 1 kWh or more. If you’re on a budget, there are a variety of less-expensive battery systems designed for computer and telecom racks. These typically cost $1000-$2000 for 5 kWh, which is remarkably affordable compared to name-brand packs. They can be paired with an inverter and solar panel management system for $1300 more, resulting in a functional system under $5000. There’s a whole world of DiY solar and battery hackers out there, and they’re doing amazing work building batteries out of surplus laptop cells, but this is certainly not for everyone. Still, there are some amazing resources out there like Will Prouse’s mobile-solarpower.com and diysolarforum.com.

The Tesla Power Wall is a very nice integrated battery solution that works with or without solar panels. As of 2023 you can only buy a Tesla Power Wall with a complete solar solution, and the battery unit itself costs $12,850 for 13.5 kWh. Tesla uses lithium nickel manganese cobalt oxide (NMC) power cells in the Power Wall and carefully manages them (reducing usable capacity) to avoid damaging them. Tesla also sells solar roof tiles, but they are really not competitive: I priced out a Tesla solar and Power Wall solution for my home and it was over $100,000! Even a roof-mounted solar panel system with a power wall came out over $30k from a local provider.

There is also an emerging class of so-called “solar generators” which combine solar and other DC inputs with an AC inverter and battery in a single unit. Some of these are compact and portable while others are larger and heavier or even fixed in place. Technically even the Tesla Power Wall is a solar generator! Perhaps the best of breed as of 2023 is the EcoFlow Delta Pro, one of the most successful kickstarter projects of all time. It uses an advanced LFP battery and provides 3.6 kWh of power as part of an ecosystem of products. Spoiler alert: I just bought two!

From Extension Cords to Transfer Switches

One of the biggest issues for battery backup solutions is cabling. How do you connect your existing equipment to the generator or battery and how do you switch over (and back) when the power goes out? Most cheaper “solutions” rely on a nest of extension cords from the generator to the devices themselves, rolled out and plugged in by hand when the power goes out. This is especially an issue for portable generators since they must be placed outside when running! The inside/outside issue is one reason most people lack any kind of power protection and many generators remain unused even when the power does go out.

The recommended solution for generators is to hard-wire a power circuit from the pad or patio to the breaker board. This can be terminated inside as one or more electrical outlets, but this still requires extension cords or manual re-wiring inside. A better solution is a transfer switch, which cuts the connection of a whole circuit from the grid to the backup generator or battery. A cheap manual transfer switch costs about $200 but requires manual activation when the power goes out. Automatic transfer switches exist too, but they’re much more expensive, starting at $1000. And any transfer switch requires extensive electrical installation work.

High-end transfer switches are able to switch between supplies (grid, solar, battery, or generator) nearly instantly, often using an app or schedule, and the best can prioritize various circuits based on demand and supply of power. For example, all circuits could be cut over to battery or generator power when the power goes out and lower-priority circuits could be cut as the power reserve is depleted. These can also dynamically allocate power, for example running on solar on sunny days and switching to the grid or battery at night. EcoFlow’s Smart Home Panel is an example of this smart home transfer switch.

What Solution is Right for You?

There are a lot of options, but it’s difficult to recommend any one solution. Solar panels are amazing, but an integrated solar setup can be costly and still won’t power your equipment through an outage without a battery backup system. Generators are cheap but most people are disappointed by the (lack of) usability in practice. It’s incredibly challenging to select and construct a system that is affordable and user friendly, leading many people to simply make do without power.

Expect to pair multiple devices to arrive at a complete solution. Even the fastest automatic transfer switch or generator inverter will have a gap, so a downstream UPS is required to keep computers up and running. And solar panels work best when paired with a battery and automatic transfer switch. You get the picture: You’re going to be buying and integrating a few different components.

Stephen’s Stance

I recently went through the process of evaluating and selecting an emergency power solution, and this article is the first in a series to lay out the various options. Although I’m something of a hacker, I decided to go with an integrated ecosystem rather than DiY, yet I couldn’t justify the expense of a Tesla solar roof and Power Wall setup. I ended up purchasing a pair of EcoFlow Delta Pro “solar generators” and their Smart Home Panel switch. I’ll go into more detail in later posts!

© Stephen Foskett for Stephen Foskett, Pack Rat: Taming Power Outages in the Home and Small Office

A Little Background on ADS-B

Although it has a creepy name, Automatic Dependent Surveillance–Broadcast (ADS-B) is pretty cool. Most countries now require all aircraft to broadcast flight data, including position, altitude, speed, and heading, using a standard digital packet-based protocol. This data comes from on-board GPS receivers and other instruments. Larger, high-flying aircraft send these once per second at 1090 MHz. The goal of ADS-B is to augment or even replace radar as the primary means of coordinating flights and avoiding accidents.

Essentially, ADS-B lets receivers “see” every aircraft in the sky, and this data is used by pilots, air traffic control, and many others besides. Take a look at FlightAware and you’ll a real-time visualization of ADS-B data from thousands of receivers all around the world. But this data isn’t just collected and used by companies like FlightAware: A network of independent enthusiasts is also collecting, processing, and sharing this data. This was exemplified by ADSB Exchange, but after they were purchased by a for-profit company the task was picked up by ADSB.fi and others.

As mentioned, all aircraft are required to broadcast ADS-B data, including private and military planes. But FlightAware and similar services only focus on commercial traffic, filtering out everything else. Wouldn’t it be interesting to see everything?

Software-Defined Radio and the ADS-B Community

A few years back, cheap USB TV tuners began appearing on the market. But hardware hackers quickly noted that the RealTek RTL2832U ASIC could receive much more than TV data. Indeed, custom software could enable this little chip to receiv data across the spectrum, including weather, emergency services, and even satellites. A whole world of hobbyist software-defined radio (SDR) applications was opened up!

It’s important to note that this chip is receive only. In other words, it can “listen” to nearly any frequency but can not transmit. There is no risk of interfering with fire fighters, astronomers, or air traffic controllers. And the USB device needs software and supporting hardware to function.

Thankfully, the ubiquitous Raspberry Pi has plenty of horsepower to support most SDR applications, and free software is widely available. Applications like Dump1090 can extract and decode data transmissions from the RTL SDR device, making ADS-B data widely available and usable. This has been embraced both by FlightAware and by a broad international community. There are literally tens of thousands of independently-operated ADS-B receivers collecting and sharing this data all around the world.

Build Your Own ADS-B Receiver!

If you’re interested in exploring this community, it’s easy and cheap to build your own ADS-B station, using an RTL-SDR dongle and a Raspberry Pi. It literally took less than half an hour for me to start receiving data and sharing it with FlightAware and ADSB.fi! I’m going to list the simple steps here, but I encourage to to get involved with the community if you want to go deeper or try a different solution.

What you need:

• A Raspberry Pi – including appropriate power supply and SD card (I used a Model 3B that I had laying around)
• An RTL-SDR dongle – I bought a kit from rtl-sdr.com and had it shipped direct from China but it’s also available from Amazon (watch out for fakes!)
• An antenna – The RTL-SDR kit I bought came with an antenna that works fine
• Software – Raspberry Pi OS Lite, PiAware, adsb-fi-scripts (there are a lot of software options but this works)

What to do:

1. Set up the Raspberry Pi
   • Install Raspberry Pi OS Lite (bullseye 32-bit) on a micro SD card
   • In raspi-config, configure localization, enable SSH, and set up Wi-Fi
   • Install fail2ban, add your ssh keys, and disable password-based and root login
   • I strongly recommend using an official Raspberry Pi power adapter, since generic ones often lack sufficient power and you’re going to have a juice-sucking USB device permanently attached and running!
2. Connect the RTL-SDR dongle and antenna
   • Plug the dongle into any USB port on the Pi
   • Screw the antenna base cable to the dongle
   • Attach the long antennas to the antenna base
   • Arrange the antennas in a “T” shape but do not extend them further
   • Attach the antenna base to the tripod and arrange it so the antennas are vertical (pointing up and down)
   • Place the antenna near an outside wall, as high up as possible
3. Set up the ADS-B software
   • Download and install the piaware-repository, dump1090, and dump978 per the instructions from FlightAware
   • Reboot and you’re in business! Go to raspberrypi:8080 to see a web interface on your device!
   • Register for a FlightAware account and claim your PiAware instance to get a free Enterprise account
   • Install the adsb-fi-scripts per their GitHub to also send data to ADSB.fi
   • Install their local interface too (step 4 in the GitHub) and visit raspberrypi/adsbfi/ to see their graph

You will be tempted to stretch the antenna all the way out and arrange it like a “V.” Don’t do this. Antennas are bizarre witchcraft. A quarter wavelength antenna at 1090 MHz is just 65.3 mm (2.57 inches) and a half wave (twice this) is roughly the length of the longer metal antenna in the kit. That’s the longest antenna you want! And it’s best to orient it straight up and down (like a sideways “T”). Just do this for the time being until you’re ready to nerd out about the antenna.

I love that the ADSB.fi interface generates a coverage map for you, as seen at right. It’s a rough indication of how far your rig can see, and is regenerated every time you boot. I recommend setting everything up and letting it run for a few hours in various spots around the house to fine the most convenient and effective spot. I did exactly this and my ADS-B station now covers most of Northeast Ohio! I’m tracking 13 aircraft and receiving over 80 messages per second as I type this.

What’s The Point?

Although I was excited enough about the prospect of my own ADS-B receiver to but the SDR dongle and set it up, I can imagine some people might be confused. What’s the point of tracking planes? I gave that a thought:

1. The hacker ethos is all about seeing the unseen and investigating the unknown. It’s fun to slap together a few bits of hardware and software and have it do something unusual.
2. Because ADS-B is not encrypted or authenticated, some have suggested it could be spoofed or hacked. A proliferation of independent ground stations and robust data collection and processing can help detect or prevent this, keeping everyone safer.
3. If you’ve ever heard a plane and wondered what it was, nothing beats having your own receiver! Within 12 hours I spotted (and heard) a Canadian CC-130, an American E-6 Mercury, and a rescue helicopter, and none of these were listed on the commercial flight tracker apps!
4. Nerds love radio, free software, and airplanes and this brings it all together.
5. Hey, free Enterprise-level FlightAware account?

Yes, FlightAware is a for-profit company and they are selling the data I’m collecting. But I’ve used their data for free for years, and am ok with this trade. Plus it’s simple to contribute to the open community at ADSB.fi at the same time!

Stephen’s Stance

This whole process took about 30 minutes from unpacking to planespotting, plus a few hours tinkering with antennas. It’s a great way to explore the world of software-defined radio (SDR) while contributing a useful data source. Plus it’s fun to know what that plane was!

© Stephen Foskett for Stephen Foskett, Pack Rat: Monitor Air Traffic With a Software-Defined Radio and a Raspberry Pi

You would probably like my other article, How to Connect Everything from Everywhere with ZeroTier

FreeNAS, TrueNAS, and ZeroTier

TrueNAS is one of the best options for building a home or small office fileserver. Formerly known as FreeNAS, it uses ZFS and FreeBSD to provide reliable and flexible storage. I’ve previously written about my home TrueNAS build, which I use for general file storage as well as Time Machine backups, and it’s still going strong.

ZeroTier was offered as an installable package for FreeNAS but was removed in version 11.3. According to Kris Moore, SVP of Engineering at iXsystems, maker of TrueNAS, it was removed for licensing reasons. Apparently, ZeroTier no longer allows governmental or SaaS use, and iXsystems felt this was incompatible with TrueNAS. Although it is possible to access a TrueNAS system using OpenVPN, WireGuard, or even through a router on a ZeroTier network, I strongly prefer a native ZeroTier client.

Install FreeBSD Packages on TrueNAS 12

Since TrueNAS is built on FreeBSD 12, it can easily run the native FreeBSD build of ZeroTier. Although TrueNAS is configured by default not to allow installation of FreeBSD packages, it is fairly easy to enable this and install ZeroTier.

The first step is to allow FreeBSD packages to be installed. As documented by Justin Silver, this requires modifying two files to enable access to the FreeBSD repository. SSH to your TrueNAS box and do the following.

Disable local packages:

sudo vi /usr/local/etc/pkg/repos/local.conf
# Change "enabled: yes" to "enabled: no" to turn off access to the local packages repo

Enable FreeBSD packages:

sudo vi /usr/local/etc/pkg/repos/FreeBSD.conf
# Change "enabled: no" to "enabled: yes" to allow access to the FreeBSD packages repo

Now you will be able to install any FreeBSD package using the native pkg utility. This will not persist following a reboot, so updating ZeroTier won’t be possible without going through these steps again!

Install ZeroTier on TrueNAS 12

Assuming you have already set up ZeroTier and have a network ID, type the following in your TrueNAS terminal to install and configure the FreeBSD ZeroTier package:

sudo pkg install zerotier
sudo /usr/local/sbin/zerotier-one -d
sudo /usr/local/bin/zerotier-cli join <your network ID>

If all goes as expected, your TrueNAS box will now show up in your ZeroTier Central interface. Accept it and (optionally) manually assign an IP address.

Use ifconfig to validate that the interfaces are up, and type sudo /usr/local/bin/zerotier-cli listnetworks to ensure that any networks show “OK”. Next, test connectivity by pinging your TrueNAS machine from another ZeroTier client and vice-versa. If everything went OK you should have no trouble accessing your file server from anywhere!

Make ZeroTier Persistent on TrueNAS 12

But there’s still one big issue with TrueNAS: It wipes all this configuration after every reboot! So you’ll have to re-start the ZeroTier daemon and re-join all your networks each time you reboot. That’s a bummer.

If you want to make ZeroTier (or any other service on TrueNAS) persistent across reboots you have to do some black magic to override this behavior.

The TrueNAS developers really don’t want people mucking about with FreeBSD, and for good reason. Making changes to the OS is a great way to ruin a perfectly good system install! But it is possible to modify the configuration in a way that persists, and that’s what we’re going to do. Proceed with caution!

ZeroTier uses a few components in different places, some of which won’t be touched by a TrueNAS reboot. The binaries in /usr/local/sbin and /usr/local/bin are safe, but the configuration files in /var/db/zerotier-one and the daemon setup won’t survive a reboot.

We need to add a ZeroTier script to /etc/local/rc.d and add an enable to /etc/rc.local so the daemon will start on boot. Thankfully, ChanceM has put in the work for pfSense (which is also FreeBSD based) and we can use his zerotier rc script!

# Go to https://github.com/ChanceM/pfSense-pkg-zerotier/blob/master/zerotier and copy the text of that file

sudo vi /etc/local/rc.d/zerotier
# Paste in the text and save with :wq

# Make it executable
sudo chmod 555 /etc/local/rc.d/zerotier

# Test your new ZeroTier service
sudo service zerotier stop
sudo service zerotier start

Now we’ll set FreeBSD to start the ZeroTier daemon automatically on reboot.

sudo vi /etc/rc.conf

# add the following line right before nginx_enable:
zerotier_enable="Yes"

Before you proceed, check that your ZeroTier configuration is 100% working!

TrueNAS stores the persistent configuration in /conf/base and uses this to wipe the /etc, /mnt, and /var directories on boot. You can modify /conf/base and make changes to the config using the following commands.

# Before proceeding, get ZeroTier installed and join any and all networks you want to persist!

# back up the ZeroTier components that need to persist
cd / ; sudo tar -cvf ~/zerotier.tar etc/rc.conf etc/local/rc.d/zerotier var/db/zerotier-one
# make the root filesystem writable
sudo mount -uw /
# restore our configuration to the persistent area
cd /conf/base
sudo tar -xvpf ~/zerotier.tar

That’s it! ZeroTier should persist across a reboot!

It’s also likely that upgrading to a new version of TrueNAS will wipe out some or all of this configuration, but the zerotier.tar file we created above should be a good backup. But be careful because restoring it will blow away the contents of rc.conf and that could be bad!

Time Machine from Anywhere with ZeroTier and TrueNAS

My next step is to enable Time Machine backups from my various Mac machines to my TrueNAS box. I created a second ZeroTier network for this so I can have better control over the process: Time Machine starts automatically when you are “on network” with the target server but this isn’t always desirable when you’re connected via LTE or other expensive or slow networks. Since it’s simple to have multiple ZeroTier networks configured and running at once, I can connect to or disconnect from my Time Machine network to allow backups to happen.

Once the Time Machine network is set up, I connect to the fileserver using the ZeroTier Time Machine network IP address and configure Time Machine on my Mac to use this as a target. After this, all I have to do is connect to the Time Machine network in ZeroTier and my Mac will see the file server and begin backing up on a regular basis!

I actually haven’t gotten this working yet because I’m having issues backing up to TrueNAS shares. But I was able to get it working from one Mac to another, so it ought to work fine. The issue is TrueNAS not ZeroTier.

Stephen’s Stance

Don’t do this. Seriously, if you’re wondering if you should enable ZeroTier natively on TrueNAS but aren’t sure about all this, just stop now. I’ve been a UNIX sysadmin for over 25 years, and I’m extremely comfortable mucking about with systems, but most people probably aren’t. Also I don’t really have time to debug your particular configuration, so please don’t expect this of me.

If you’re a UNIX nerd like me, though, this is an effective way to install ZeroTier natively on TrueNAS. It’s not perfect (the ZeroTier interfaces don’t show up in TrueNAS Network Summary, for example) but it’s definitely usable! And maybe all this will help others wondering how to install software and persist across reboots on TrueNAS.

© Stephen Foskett for Stephen Foskett, Pack Rat: How To Install ZeroTier on TrueNAS 12

Connectivity Challenges of the Modern Nerd

I have a variety of computer systems I like to access, and most are inherently inaccessible. Some of this is incidental, thanks to NAT and consumer telcos, but I’m also careful to lock them up with firewalls and access controls. I’ve been searching for a reliable, friendly, and secure way to access everything from everywhere since forever, and I finally have just the thing up and running!

Here’s a rough inventory of the systems I’m trying to access, and why it’s hard to get to them:

• My home network is served by two unreliable consumer ISPs (Spectrum and Windstream), each with its own NAT. These are fed to a pfSense router with another layer of NAT, but I often switch ISPs as one falls on its face. Inside my home I have a firewalled VLAN for IoT devices and another for my home lab. I’ve also got a big TrueNAS file server, some Macs, a few Ubuntu machines, and a Home Assistant instance.
• My office has amazing high-speed connectivity but the building router is inaccessible to me and feeds my office pfSense with two layers of NAT. Here again, I’ve got a firewalled VLAN for my office lab machines, along with a bunch of Macs, a file server, a few more Ubuntu machines, and another Home Assistant instance.
• My web servers run in Linode using Docker and are well firewalled.
• My mobile clients include my new 14″ M1 MacBook Pro, my trusty iPad Pro, and my iPhone. I’m often connected using mobile data of course, but I also sometimes use Wi-Fi at hotels, family, or work spaces.

The challenge, put simply, is how to be able to connect to my home network, my office, and my web servers from any of my mobile clients, regardless of the connectivity available at each location. And how to keep it all secure so my remote access doesn’t let others get in and mess up my stuff!

By far the biggest issue is all that NAT. If you’re not familiar with the issue, network address translation (NAT) “hides” a private network behind a single IP address, dynamically routing connections to the right client. This is why so many home and office routers use IP addresses beginning with 192.168 or 10: They’re NATting this private network. The problem is that some services (especially if they deal with incoming connections) need a public IP address or at least a public port. But some routers don’t “do” UPNP port assignment right, and even if they do you still need a dynamic DNS provider to deal with dynamic IP addresses. Remote access behind NAT is a mess, and it’s even worse when you have NAT behind NAT and no access to any of the routers!

NAT, DNS, and firewalls also pose a serious challenge on the client side. It’s sometimes hard to get back to an open port since some providers block access to well-known VPN ports or IP addresses or force the use of their own DNS resolver or other client “services” that muck up the works.

There are many remote access services and VPN providers, but these don’t solve the issues completely. It’s hard to set up a VPN server behind a NAT in the best circumstances and nearly impossible behind double NAT, especially if you don’t control the router. And remote desktop solutions tend to have limited client support, over-reliance on UPNP, or are system resource hogs.

Up to now, the only moderately-reliable connectivity scheme I’ve had was a combination of OpenVPN and SSH. Because I was able to control my home routers, I could set up an OpenVPN service on my home pfSense, punching through both NATs with a private port and using a dynamic DNS provider to locate the IP. Then I set up an OpenVPN client at work, which connected outbound through double NAT to my home network. I provided access to my client devices using another OpenVPN server and OpenVPN clients for iOS and macOS. I provided backup access using an SSH endpoint at home on a high port. This solution worked, but was incredibly finicky. And since the world can see my dynamic DNS hostname, I had to be very careful to secure it.

ZeroTier Changes Everything

Once I started playing with ZeroTier, I realized that it could do everything I wanted and more.

The most powerful aspect of ZeroTier is that it is designed to punch through NAT, firewalls, and dynamic network infrastructure from the inside out. There are no static accessible servers – everything is a client and everything connects to the ZeroTier service before turning around and accessing your other resources over a private network.

And rather than playing around with high-level services, ZeroTier connects everything with a simple virtual layer-1 and 2 Ethernet network. It provides some higher-level services (IP addresses and DNS) but it doesn’t even pretend to be a remote access service. It’s a virtual network connection, and you can run anything on top of it.

It’s also incredibly robust and secure, with end-to-end 256-bit encryption and a “zero trust” design so your network is your own. And it runs on pretty much everything, from Linux to macOS and iOS to Windows and Android. It’s even available as a plugin for Home Assistant, Mikrotik/OpenWRT, and many NAS operating systems! It’s not yet robustly supported by TrueNAS or pfSense, but I expect that will come with time.

I was able to get ZeroTier up and running in just a few minutes, and it provided robust and reliable access to everything while I was out of the country on dodgy internet for a month. I was even able to set up a private “home run” back to the USA so my Apple TV worked! In fact, ZeroTier was the most reliable part of my infrastructure over this time, punching through a triple-NAT-quadruple-Wi-Fi-Repeater sandwich and quickly coming back online through reboots and outages from various other components.

ZeroTier is not perfect, however. It is unapologetically nerdy, with heavy reliance on CLI commands and an assumption of network and system administration competence. And the new DNS feature really needs work. But even with minimal documentation I was able to get everything up and running, and I expect my readers could do the same!

One of the criticisms of ZeroTier is that, although it is fundamentally a peer-to-peer network, it relies on root servers operated by the company to enable connectivity. But this is really a philosophical rather than a technical issue, since nearly every service relies on some root nodes, from DNS to the internet itself, and ZeroTier actively tries to “get out of the way” by encouraging direct peer-to-peer connectivity for performance reasons. Additionally, the system is designed with “zero trust” end-to-end encryption, so your data is isolated from any other users of the network.

Operationally, setting up a ZeroTier network is incredibly simple:

1. Create a ZeroTier account on ZeroTier Central
2. Provision your own network using their web interface, including setting IP address ranges
3. Install the ZeroTier One client on any device and enter your network ID
4. Accept the client into your network using the web interface and configure an IP address for it

Once a client is added to your network, you can access it from any other network client as if they were on the same LAN. It’s really that easy, and in my experience works through anything and from anywhere!

More advanced users can set up DNS or routing within their ZeroTier network, allowing it to act as a full-blown VPN. But even without these features enabled, the VPN-like iOS and macOS client enabled me to use my servers and access my systems using IP addresses.

Note: ZeroTier is presenting at Networking Field Day in January, 2022! Tune in Thursday, January 27, at 1 PM Pacific time for a technical deep dive into ZeroTier with some of the smartest people in enterprise networking. We’ll also post recordings of the session on the Tech Field Day YouTube channel shortly after. Although this is my company and event, I “discovered” ZeroTier independently and this article was not sponsored or influenced by this sponsored event in any way!

Stephen’s Stance

It is rare that I am so enthusiastic about any service, but ZeroTier does exactly what I needed. With very little effort, I was able to access the key components of my infrastructure from any device, regardless of network configuration or availability of IP addresses or ports. And everything I’ve tried to do (from the Home Assistant UI to SSH to SMB file transfer) has worked flawlessly and efficiently. Apart from the fidgety user interface, ZeroTier is the answer to my connectivity prayers!

© Stephen Foskett for Stephen Foskett, Pack Rat: How To Connect Everything From Everywhere with ZeroTier

Read my entire Rabbit Cloud series

• Introducing Rabbit: I Bought a Cloud!
• Running Rabbits: More About My Cloud NUCs
• Tortoise or Hare? Nvidia Jetson TK1
• Powering Rabbits: The Mean Well LRS-350-12 Power Supply

The Rabbit Power Design

The “Rabbit” trays have a simple power design, with two of three V+ rails of the power supply going to a pair of 10-fuse blocks before being distributed to the various devices. The V- rails are split and sent directly to ground on the devices. It’s a workmanlike but unbalanced design, and surprisingly is not wired uniformly rack-to-rack.

The Mean Well LRS-350 is part of a series of similar slim power supplies from the Chinese company. The LRS-200 models are passively cooled and max out around 200 Watts, while the LRS-350 have a fan and can go to 350 Watts. The supply can handle 90-132 volts or 180-264 volts, switchable via a switch on the side. Input and output is routed to a block of nine screw terminals on one side, along with a trimmer and LED.

The power distribution design essentially splits the +12V output into two, each with a separate fuse block. One powers five NUCs and five Jetsons, while the other powers the other five NUCs and the Ethernet switch. The -12V (ground) blocks are split in three: One for five NUCs, one for five Jetsons, and one for five NUCs and the switch. The fuse blocks are numbered 1-10, but each of my trays is wired differently, with some putting the first fuse on the nearest NUC and others putting it on the farthest. This lack of uniformity really bothers me, but it works fine.

Although all three devices (NUC, Jetson, Ethernet switch) use 12 volt power, each has a different barrel connector. The trays have the proper connector for each device and all the wires are neatly routed and zip tied in place. Note that all three devices can use the same barrel connector, but it’s not a perfect fit. It’s nice to see that they specified the right one. This makes me feel a little better.

The fuse blocks use “Micro 2” 3 Amp fuses. These appear to be fairly hard to find at local stores, so I’m ordering a pack of them. For now I’ve been swapping fuses as needed from the six blocks on three racks. Since a few of my NUCs arrived DOA, I’ve got spaces. I have blown one fuse already, thanks to a short when plugging in a Jetson. It’s good to know that they work.

Risk of Electric Shock

The exposed screw terminals present a safety risk, and could potentially short out if something is dropped on them. So I immediately insulated them in electrical tape while considering how better to cover them. I decided to design a 3D printed cover for the terminals (and the entire end of the PSU) to better protect myself and my Rabbits. Download my Mean Well LRS Power Protector from Thingiverse.

It seems that the Mean Well LRS is popular with the DIY and tinkerer crowd, so there are a ton of other printable add-ons for it. Mine is pretty simple and small, but I may end up using a different one depending on what I do with my Rabbits.

I just used electrical tape on the fuse blocks for now, since the primary connector is exposed. At least the rest of the connections are properly insulated.

The Fan Sucks and Blows

Next, my attention turned to the power supply. The fan sucks (forces air into the power supply), but all three fans on my trays were seized up. So I guess you could say the fans also blow. These are cheap 60mm by 15mm fans with a simple 2-wire (off/on) connector.

I purchased a slightly-nicer fan from eBay to replace the bas ones. The new one is lower-profile (10mm) so it doesn’t have to work as hard, given the tight dimensions inside the power supply. It also blows a little more air than the stock fan.

Note that the fans were wired differently so I had to swap the red and black connectors to make the replacement fan work. Also, pay attention to the airflow direction: You want to have air flow into the power supply, not out as might be expected.

Once I replaced the fans, the power supply is much cooler. It was quite hot under load without the fan, and now it’s merely warm. I also designed my terminal block cover to preserve as much airflow as possible, since the end is an active exhaust. Since the similar LRS-200 is designed for passive airflow, I imagine it would be acceptable to remove the fan entirely as long as it’s not running under much load. And many of the 3D printed options include a larger fan.

Note that the LRS-350 includes a thermostat to turn the fan on or off based on internal temperature. I verified this by varying the load on the power supply and blocking the vents. The latter caused the fan to turn on, while the former did not. Also, I have found that 10 NUC5 Rabbits under 100% CPU load is not enough to kick the fan on, but 10 NUC6 Rabbits is!

Stephen’s Stance

The Rabbit trays use a standard power supply and have a well-designed power distribution design with fuses and proper connectors. Although the fans died, it was a simple matter to replace them. And my 3D printed cover helps prevent electric shock. My Rabbits are really running now!

© Stephen Foskett for Stephen Foskett, Pack Rat: Powering Rabbits: The Mean Well LRS-350-12 Power Supply

Read my entire Rabbit Cloud series

• Introducing Rabbit: I Bought a Cloud!
• Running Rabbits: More About My Cloud NUCs
• Tortoise or Hare? Nvidia Jetson TK1
• Powering Rabbits: The Mean Well LRS-350-12 Power Supply

Meet George Jetson

The Jetson was one of the first high-profile AI prototyping boards released, and was quite impressive back in 2014. This was before the Raspberry Pi 2 was released, so it would have been compared to the original Pi, which had a 700 MHz single-core Broadcom BCM2835 CPU. The TK1 sported a quad-core Tegra K1 SoC (which used the Cortex A15 microarchitecture and actually included a fifth low-power “companion” core) running at up to 2.3 GHz, blowing the doors off the Pi and actually quite competitive with low-end PCs of the day.

But this performance was overshadowed by the inclusion of a then-modern “Kepler” GPU with 192 CUDA cores. The Jetson TK1 became the darling of the AI set, with robots and autonomous driving experiments built around it. No doubt this is also why the original Rabbit streaming service specified five Jetson TK1 boards as companions to the ten NUCs on each tray. It would have been an impressive-sounding setup for the time.

But times change.

The Jetson TK1 is a surprisingly limited board today, since it’s so proprietary and integrated:

• 2GB of memory might have sounded fine in 2014, but since it’s soldered to the board and not socketed like the NUC it is a major limitation today
• The USB 3.0 port is nice, but there’s only one and it’s flaky, needing special firmware and causing issues sometimes; you can get an adapter to use the micro-USB port as a second port, but a USB 3.0 hub is easier to find
• The Gigabit Ethernet port is just slow, only able to hit 68% of the throughput of the equivalent port on the NUC
• It can boot Linux for Tegra from the 16GB integrated eMMC storage, but this was never updated past Ubuntu 14 and it’s fiddly to get anything else to boot
• The Kepler GPU is not supported by the latest software, nor is it all that fast by today’s standards

On the plus side, there are some nice aspects to the board:

• The Tegra K1 CPU is reasonably powerful for an embedded board even today
• Integrated RAM and eMMC mean the salvage company delivered a bootable and usable board, unlike the NUC which came without RAM or storage
• The mini PCI-E slot is half-length but extends “off the board” without obstruction so it should be able to accommodate other peripherals
• It has a nicer SD slot than the NUC, with a spring-loaded ejector
• It has a SATA port, though a drive would require external power

Objectively weighing the plusses and minuses of the Jetson TK1, I decided that it would be useful in a supporting role, while the NUCs were the star of the show. I planned to use a few Jetson boards on each tray as a simple router, DHCP server, DNS server, NTP server, and perhaps a storage server or cluster.

Booting the Jetson TK1

The embedded eMMC storage is the logical boot medium for the Jetson TK1, so that was my focus from the start. It could theoretically also boot from SD card, USB drive, or SATA, but all of these require a functioning eMMC pre-boot environment.

The Jetson TK1 has a micro-USB port that allows the eMMC to be flashed with Linux for Tegra or another operating system. I found the flashing process to be simple once I got used to the quirks. Rather than rehashing the whole process, here’s my quick list of lessons learned to help anyone else:

• Linux for Tegra must be installed from an Intel x86 host running Ubuntu 14.04. It will not work from a Pi, a Jetson, or a different or more modern version of Linux. You have been warned!
• Download Linux for Tegra from Nvidia’s Tegra archive. I suggest grabbing the original Release 19.3 as well as the latest Release 21.8. Get the Driver Package and Sample Root Filesystem for your release of choice. That’s all you need.
• Connect the Jetson TK1 to your Linux server with a quality micro-USB to USB cable. Some random cables in my drawer did not work. If you don’t see “Nvidia” in lsusb on the host, try another cable.
• Enable eMMC flash mode by holding the “Recovery” button while tapping the “Reset” button once. No need to hold it longer or time anything.
• Carefully follow the Quick-Start Guide, including running the commands with sudo. Make sure you have plenty of high-speed storage available; it will not fit in a 32 GB card.
• Maximize the available eMMC space by flashing using “sudo ./flash.sh -S 14580MiB jetson-tk1 mmcblk0p1”
• Linux for Tegra includes a boat-load of useless software, so clean it up after installation by running a ton of “apt-get purge” commands (I scripted these and might share that later)

Follow my lead, and you’ll have your Jetson TK1 running Ubuntu 14.04 in no time. But then what? Ubuntu 14.04 reached the end of the LTS window on April 30, 2019 (even before L4T 21.8 was released) but still works fine. Just don’t expect many bug fixes, and be prepared to “fight” to get modern software installed. I’ve already been hunting down back-ported libraries (libssl, for one) and I’m not doing much with it. And don’t even think of exposing a Ubuntu 14 server to the open internet, since there’s no telling if there are un-patched exploits out there and they likely won’t be fixed.

Hare or Tortoise?

The Jetson seems pretty snappy when running, so I was feeling good about using mine as “Hares” to support the NUC “Rabbits” in my cloud cluster. But things have not worked out well so far.

The first thing I wanted to do was configure a Jetson as a simple router and DHCP server, allowing the cloud to exist independently of the network to which it is connected. Like most people, I use 192.168 behind my NAT, so I planned to create another 192.168 subnet for each cloud tray. This would give the NUCs plenty of IP addresses even if they need a few for various containers and services.

The ISC DHCP Server is easy to set up in Ubuntu 14 and seems to work fine with the Rabbit configuration.

Since the Jetson TK1 has a single Ethernet port, I planned to use a USB 3.0 Ethernet adapter as a second port for routing. The included NetGear switch is un-managed, which makes tagged VLANs a little more challenging. My old Cable Matters ASIX-based USB 3.0 adapter was recognized by Ubuntu with no special software to install.

Routing can be as simple as enabling net.ipv4.ip_forward in sysctl.conf, especially if a firewall or NAT is not needed. It was more challenging to enable a second subnet on my pfSense router, which needed both a gateway and static route added. I also needed to add outbound NAT rules on the pfSense router, and I still don’t have this working correctly.

I am actually considering using OpenVPN to set up an outbound tunnel from the Rabbit router to my pfSense router instead. This would allow me to set up the Rabbit tray anywhere (home or office) and have it come up correctly regardless. We’ll see when/if I get around to this!

But right now I’m having major issues with the TK1…

First, it appears that the Gigabit Ethernet port is connected to a slow I/O line somehow. Some have suggested that it’s on the USB interface (like the Pi) but it appears to be on the PCI bus in lspci. Regardless, it’s slow. I recorded between 600 and 640 Mbits/s using iPerf, and it appears to be about the same in both directions. For comparison, the NUCs are pushing about 940 Mbit/s in the same test.

Doing some research, I came across this popular blog post suggesting that the open source r8169 driver isn’t appropriate for the older Realtek 8111 ethernet controller, and that one should use the r8168-dkms driver instead. Sure enough, the Linux for Tegra install does use the R8169 driver.

Although the correct driver does appear in the Linux for Tegra universal repository, I was unable to get it to “take over” for the port. After un-commenting the “universal” lines from /etc/apt/sources.list and updating the apt repository, I installed the new driver with “sudo apt install r8168-dkms”. I then blacklisted the r8169 driver per step 3 in that blog post and rebooted. But no matter what I did, the port continues to use the r8169 driver.

Even worse, I’m having issues using the Jetson as a router. Even without iptables running, it’s slow and flaky. I’m getting time-outs on ssh sessions and if I can’t reliably log in through the Jetson, it probably won’t work well for regular use. I might try setting things up using VLANs direct to the pfSense router, or sourcing a Mini-PCIe Ethernet adapter, but this seems like over-kill. Maybe I’ll just use one of the NUCs as a router.

Stephen’s Stance: Jetson TK1 is Meh

All this is to say that I am not impressed by the Jetson TK1. The hardware is inflexible and limited, the operating system is out of date, and the networking is slow. I’m frankly not sure if I will be able to use the Jetson boards for anything productive.

© Stephen Foskett for Stephen Foskett, Pack Rat: Tortoise or Hare? Nvidia Jetson TK1

Read my entire Rabbit Cloud series

• Introducing Rabbit: I Bought a Cloud!
• Running Rabbits: More About My Cloud NUCs
• Tortoise or Hare? Nvidia Jetson TK1
• Powering Rabbits: The Mean Well LRS-350-12 Power Supply

My NUC Census

I purchased three “trays” of ex-rabb.it encoding hardware from eBay. Each included 10x Intel NUC single-board computers, 5x NVIDIA Jetson TK1 single-board computers, a Gigabit Ethernet switch and cabling, and a power supply. Although most purchasers received only NUC5 machines with “Silvermont” Pentium N3700 processors, one of my trays included NUC6 machines with “Goldmont” Celeron J3455 processors. The latter also support dual-channel memory (and have two SO-DIMM slots) up to 16 GB.

This last point is a bit of surprise since Intel’s documentation for the NUC6CAYH specifically says it’s limited to 8 GB of RAM. But when I loaded it with two 8 GB DIMMs, the BIOS recognized the RAM and it was available to the Linux kernel.

Another notable difference between these machines is the BIOS. The NUC6 has many more options, especially when it comes to power. One interesting one is “Power Sense”, which will apparently throttle the CPU if the board is going to draw more than the rated power supply level. But I imagine that the Rabbit fuse block will do that too! And frankly it’s not likely I would run into this, with a 10 Watt TDP CPU and no hard disk or peripherals.

Speaking of power, I’m actually using a simple solution to run them in testing. If you’re read my blog much, you know that I’ve often “shucked” external hard disk drives for the SATA drive contained within. I’ve got a pile of these extra “wall warts” in a box, and was pleased to find that they’re just fine to bring up a NUC or Jetson for testing. They supply 12 Volts and the barrel connector is configured correctly with a positive tip. Although they’re rated at just 1.5 Amps, this is plenty of power in practice.

The highest power draw I saw when running Geekbench was just 0.20 Amps (12.8 Watts). Trying to generate more power draw, I fired up the xmrig Monero miner with four threads. Although the CPU is running at maximum speed (2.3 GHz) the fan is barely pushing any air and gets only warm to the touch even after hours of running. These little “bunnies” are amazingly power efficient!

And while we’re on that topic, you can put any thoughts of a massive Monero miner out of your head. Neither Atom CPU has enough L2 cache to run the RandomX algorithm efficiently, so it has to go to main memory constantly. The Pentium N3700 can complete 280 hashes per second, while the Celeron J3455 hits 382 hashes per second. Dual-channel RAM in the Celeron allows it to hit 420 hashes per second with two RAM sticks. This is small potatoes: The Core i3-5010U in my NUC5 can do 640, my Core i7-6700K hits 1800, and my new Ryzen 9 3900X is steady at 9975. Or to put it another way, all 30 NUCs can’t match the hash power of a single Ryzen 9 CPU, and they would use twice as much power doing it!

How to Boot? That is the Question!

The NUC5 and NUC6 can boot from a variety of media:

• PXE Network Boot (this is how they come from rabb.it)
• SATA (requires a powered drive or custom cable)
• USB disk drive (draws more power, expensive)
• USB flash drive (iffy reliability, cheap)
• SD card (not the fastest, cheap)

I’m sticking to the latter for now. I definitely don’t want to mess around with PXE network booting, so that’s right out. Then there’s the issue of sourcing SATA cables and SDDs at a reasonable price. And external USB disk drives don’t sound that attractive. So I’m sticking with SD cards for now.

After browsing Jeff Geerling’s SD card tests, I’m using SanDisk Extreme and Samsung Evo Plus cards. My own experience has shown me just how unreliable and slow generic SD cards can be, and how many counterfeit and defective cards there are in the e-commerce supply chain, so I’m paying a little more for name-brand cards from major retailers.

Best Buy had the best price on the Samsung cards so I bought a bunch of them at retail, and boy are they fast! Plus, SD cards support TRIM out of the box, unlike most other media, and have gotten much more reliable. Short of a real SSD, booting from SD seems like a winner.

Installation of Ubuntu went pretty smoothly once I got started with the SD cards. This is no surprise, since these are basically little PCs with well-supported hardware. This is one reason the NUCs are so useful: Even though they use an oddball Atom CPU, they’re not really unusual at all. I expect that they will be useful and supported for years, if not decades to come. The same can not be said of the Jetson TK1, as I will discuss in my next article!

I created a generic install of Ubuntu 18 on one of the Rabbits and copied the card image to my Mac using dd. Now I can create more copies whenever needed and only have to do a little tuning to get the next Rabbit running: Change the BIOS to disable PXE, enable boot from USB, and enable USB port 4 (which is mysteriously disabled in BIOS on every NUC) and set the hostname to a generic value based on its position in the Rabbit tray.

Note: I experienced some “weirdness” getting the Ubuntu Server install process to recognize an SD card or a USB flash drive as a valid install target. I am not sure if I did something wrong or if Ubuntu’s latest installer really doesn’t want to work with them. But the older images and installers seem to work fine. I solved this by putting the SD card in an external USB reader, and the installer happily wrote to it.

Stephen’s Stance: Good Little Rabbits

My experiments so far have cemented my opinion on the value of the NUCs in a cloud lab environment: They’re plenty snappy with 8 or 16 GB of RAM, even using an SD card for storage. The tray power supply is efficient and capable, and the whole tray doesn’t make much noise or heat. I’ll write more about that soon! Overall, with just about $1k invested, this “cloud of rabbits” is working out fine.

© Stephen Foskett for Stephen Foskett, Pack Rat: Running Rabbits: More About My Cloud NUCs