Mini-PC OPNsense Firewall

2024-09-28T20:55:00.020+10:00

Introduction

Here I summarise the install of a dedicated firewall for home use with 'NGFW' (Next-Generation Firewall) features and analytics made up of a combination of OPNsense, CrowdSec and Zenarmor.

History

From 2005 onwards I started using mini-PCs as a network appliance based on Mikrotik's RouterOS installed onto PC Engines WRAP boards supplied by Yawarra Tiny Computers. You can read about them here: https://blog.naturalnetworks.net/2005/08/pc-engines-wrap.html .

Back then these mini-PCs were consuming around 8 to 10 watts after including a wireless adapter. They weren't exactly powerful but could accommodate the bandwidth I was dealing with at the time (<100mbit/s) without issue and didn't need cooling for the CPU. These days a modest mini-PC consuming the same power is 10x more powerful in every regard.

Earlier I had used Sophos XG running on an Intel 5th generation i5 NUC to gain experience with it. It ran well until the NUC died - so I managed to find a replacement NUC board, that ran okay too, until it died... oddly after the same period of time. It has been a year or so and I miss having the added visibility and protection it provided - so time to set up a replacement.

Topology

Draytek Vigor 167 VDSL2 configured as a bridge.
Mikrotik RB4011 configured as a firewall router.
Rest of the wired network connectivity provided by a Mikrotik CSS326 switch.

As I already have a router with its own stateful firewall, NAT, and various services I aimed to configure OPNsense as a transparent firewall so that it is invisible - also known as a 'bump-in-the-wire' firewall. This way the public IP address remains on the Mikrotik and I won't need to make any changes to the existing network. Another benefit is that I can bypass OPNsense by connecting the VDSL2 bridge directly to the router.

Hardware

I settled for an Intel 12th generation N100 CPU with 4 x 2.5GbE ethernet ports, you can find a review of them here.

I didn't purchase storage or memory with the mini-PC. Instead I looked for whatever was on special at my local online suppliers. I found a 250GB Western Digital Green SN350 NVMe and 16GB Crucial DDR5 SODIMM memory.

I checked the thermal paste application between the mini-PC's processor and case/heatsink - it was pretty good but I replaced it with some fancy stuff I had laying around.

Nothing further to do once the storage and memory are installed.

Hypervisor (Proxmox)

Using a hypervisor is optional - you can install OPNsense directly on the disk (baremetal) without issue. However a hypervisor brings versatility such as snapshots and being able to run other services on the mini-PC. I decided to use Proxmox as I hadn't had a chance to try it out and it is commonly used in combination with OPNsense on mini-PCs.

I had no problems installing Proxmox VE 8.2.2 on the N100 - all the network adapters (Intel I-226V) were found. Prior to going any further I disabled the Proxmox Enterprise Repositories and added the 'No-Subscription' ones and updated Proxmox using the command line:

apt update

apt full-upgrade

Rebooting the mini PC afterwards and checked the Proxmox version which showed 8.2.7.

Instead of mapping the network adapters directly to the OPNsense virtual machine I created bridge interfaces, one each for WAN, LAN, and MGMT. Essentially two port bridges between the physical interface of the mini-PC and a logical interface of the OPNsense virtual machine (Virtio paravirtualised). This is fine for my purposes although if I needed to find more performance then direct mapping the interfaces would likely be a good place to start.

I created the virtual machine for OPNsense with the following non-default settings:

4 vCPUs (1 socket, 4 cores, x86-64-v2-AES)
10GiB memory
3 Network Devices (Virtio, mapped to the three Proxmox bridges)
32GB Disk

While 8GB of memory is plenty for OPNsense, I wished to utilise RAM disks for /var and /tmp to reduce wear on the cheap WD Green SN350 NVMe which has limited endurance specification (even with 40TBW I doubt this would be an issue regardless). So I bumped the memory to 10GB and will send logs to a NAS for retention.

I enabled power saving on the host to try and reduce power consumption (reduce cost, increase UPS runtime) and heat generation (passively cooled device). You can do so from the shell from within Proxmox:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

The output of the above in my case was 'performance' and 'powersave'. So I use powersave:

echo "powersave" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

To make these changes persist across reboots, edit /etc/crontab and add the following to the end:

@reboot echo "powersave" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null 2>&1

Next is to limit the number of disk writes, as this is only a single node (not running a Proxmox cluster) HA services which are heavy on the disk writes can be turned off. To disable the HA services using the cli:

systemctl stop pve-ha-lrm

systemctl disable pve-ha-lrm

systemctl stop pve-ha-crm

systemctl disable pve-ha-crm

systemctl stop corosync.service

systemctl disable corosync.service

Disable logging to disk by adding the following two settings to the end of /etc/systemd/journalctl.conf:

Storage=volatile

ForwardToSyslog=no

You can monitor the write utilisation of the disk using:

smartctl -a /dev/nvme0

Look for 'percentage used' if available and see if it's increasing rapidly - eg it shouldn't increase more than say 1% a month, preferably less. You may need to use the TB written and calculate it from the manufacturer's specification.

Upload the OPNsense ISO to proxmox and mounted the image to the new virtual machines 'DVD' drive.

Given that this is a passively cooled device sitting in a closet, I wanted to monitor the CPU temperatures. Proxmox doesn't provide this however you can use the a script from https://github.com/Meliox/PVE-mods to provide CPU and NVMe temperatures on the Proxmox node summary.

apt-get install lm-sensors
wget https://raw.githubusercontent.com/Meliox/PVE-mods/main/pve-mod-gui-sensors.sh
bash pve-mod-gui-sensors.sh install

I have found that Linux's 'Split Lock Detection' causes intermittent issues with OPNsense in my particular setup so I disabled it by setting the following in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet split_lock_detect=off"

The issues I found was that persistent connections would be reset, for example DNS over HTTPS (DoH) sessions would reset and have to be reestablished. Moreover, Proxmox would crash requiring a power cycle.

OPNsense Installation

As mentioned I would like to add OPNsense as a transparent firewall so as to minimise changes to my network while keeping the attack surface to a minimum as the firewall itself can come under attack or end up with its own vulnerabilities that I need to manage.

Transparent Firewall (Bridge)

Follow the OPNsense guide: https://docs.opnsense.org/manual/how-tos/transparent_bridge.html

I suggest using floating firewall rules with their state set to either 'sloppy' or 'none', in my case I only required a handful to ensure all traffic was allowed, including the firewalls traffic via the management network. The existing firewall on the router will handle the layer 4 filtering.

If using IPv6, make sure to enable 'Link-local Address' when configuring the bridge interface. Otherwise strange behaviour will occur.

The only other tweak I made was to set the State Type of the floating rules to 'sloppy state' - likely not needed however I was finding a few persistent connections from IoT devices that would continuously trip up the invalid state block until I set this.

General

A handful of configuration items needed, primarily enabling the RAM disks and sending logs to the NAS.

Set up configuration backups - 3 revisions
Configure DNS - I use a pihole running as a container on the NAS
Update to the latest firmware version
Logging - set up remote logging to the NAS
Enable RAM drives for /var and /tmp - use 10% of memory each.

NetFlow/Insight

I like the traffic insights provided by OPNsense so I enable NetFlow and collect the flows for the on-box Insights reports.

Add the WAN and LAN interfaces as NetFlow listening interfaces.
Set the WAN interface as the... WAN interface
Tick the 'Capture Local' box to begin to inform the local Insights

Zenarmor

Ref: https://www.zenarmor.com/home-security

Zenarmor provides extra-firewall intelligence feeds in the form of curated categorised lists.

Follow the OPNsense guide: https://docs.opnsense.org/vendor/sunnyvalley/zenarmor_install.html

Configuration options are limited under the free version. Much of it is self explanatory - I set the monitored interface to 'LAN' and I enabled all the 'Essential Security' items except DNS over HTTPS as I use that.

CrowdSec

Ref: https://docs.crowdsec.net/docs/intro

CrowdSec is similar to Zenarmor is that it provides extra-firewall intelligence however it also monitors the local firewall logs for malicious activity and executes scenarios to defeat it.

Follow the CrowdSec guide: https://docs.crowdsec.net/docs/getting_started/install_crowdsec_opnsense/

I use the following blocklists:

Firehol cybercrime tracker list
Firehol greensnow.co list
Free proxies list

Intrusion Detection

OPNsense comes with Suricata IDS. I enabled the service and set the monitored interface to 'WAN' so as to not conflict with Zenarmor, which is monitoring 'LAN'.

I enabled various rulesets from abuse.ch and ET Open and set the update schedule to occur daily. I then created a policy to drop any traffic that had a rule signature of 'major' or 'critical'. I find this provides a reasonable amount of protection with little administration overhead.

One key setting for when running on a CPU that provides Streaming SIMD Extensions 3 (SSE3) is the Pattern Matcher (Settings -> Intrusion Detection -> Administration, Detection). Setting this to Hyperscan allows use of SSE3 for intrusion detection, greatly increasing performance.

QEMU Plugin

As OPNsense is running on a Promox hypervisor, it's wise to install the QEMU Agent to allow the hypervisor to better manage this guest.

Install the QEMU Guest Agent plugin and set 'virtio_console_load' = yes tunable
In Proxmox, enable the QEMU Guest Agent within the OPNsense VM's options and reboot

Conclusion

With a small investment in hardware I have added smart network security controls to my home network that require little ongoing maintenance and management due to the automatic updating of the threat feeds.

Resource wise OPNsense is peaked at 7 - 8GB memory and settled at 3 - 4GB and CPU occasionally spiking to 50% (I suspect a single core or two might be spiking to 100%). Disk usage is sitting at 25% after a week.

Compared to my experiences with Fortinet, Cisco (ASA & FTD), Checkpoint, Sophos and Sonicwall firewalls - OPNsense offers a reasonable level of security coverage and features when combined with CrowdSec and Zenarmor, while still being easy to use and intuitive.

Follow up

I have added a fan to sit on top of the mini-PC as my closet gets warm with the other equipment - a 120mm case fan connected to the screw terminal directly behind the 12V DC socket has brought CPU temperatures down to 40°C from 60°C.

Natural Networks