Mi blog lah!

How to run Firecracker microVMs with vmsan on Incus

Simos Xenitellis — Sun, 08 Mar 2026 21:48:08 +0000

Incus is a manager for virtual machines and system containers.

Incus, to run VMs, uses KVM as a Type 1 hypervisor, and QEMU as a Type 2 hypervisor and emulator.

On the other hand, vmsan only uses KVM as a Type 1 hypervisor. This means that it has far less features, but it is very lightweight.

In this post we have a look at vmsan by running it in an Incus virtual machine.

Prerequisites
Setting up vmsan
Using vmsan
- vmsan create to create a microVM
- vmsan list to list the microVMs
vmsan connect to get a shell into a microVM
Running nginx in a microVM
Conclusion

Prerequisites

You need to

have Incus installed and initialized
have configured Incus to run virtual machines
have configured your system for nested virtualization

For the last two points, check out this post,

How to run an Incus VM inside an Incus VM (nested virtualization)

Setting up `vmsan`

We launch an Incus VM and in there we install vmsan. By default, the Incus VM will have 1GiB RAM and 10GiB disk space. When you run incus shell, the instance should have finished the start-up. Otherwise, you get an error as shown below. Just try again and until you get the command succeeds.

$ incus launch --vm images:ubuntu/24.04/cloud microvm
$ incus shell microvm
Error while executing alias expansion: incus exec microvm -- su -l
Error: VM agent isn't currently running  // it just means the VM did not finish yet booting.
$ incus shell microvm
root@microvm:~#

Then, we follow the instructions at https://vmsan.dev/getting-started/installation to install vmsan. First we install curl because the installation script is a curl pipe script. And we also install iptables because currently it’s not installed automatically by the installation script. This may change in the future. Note that unzip and squashfs-tools are both installed by the installation script, as shown below. We are also prompted to setup an optional Cloudflare tunnel which we do not do in this tutorial for simplicity.

root@microvm:~# apt install -y curl iptables
...
root@microvm:~# curl -fsSL https://vmsan.dev/install | bash

  vmsan installer
  Firecracker microVM sandbox toolkit
  https://github.com/angelorc/vmsan

[info]  Installing prerequisites: unzip squashfs-tools...
[ok]    Prerequisites installed
[info]  Node.js not found — installing Node.js 22 via NodeSource...
...
2026-03-04 18:15:18 - Repository configured successfully.
2026-03-04 18:15:18 - To install Node.js, run: apt install nodejs -y
2026-03-04 18:15:18 - You can use N|solid Runtime as a node.js alternative
2026-03-04 18:15:18 - To install N|solid Runtime, run: apt install nsolid -y 

[ok]    Node.js v22.22.0 installed
[info]  Setting up /root/.vmsan...
[ok]    Directories created
[info]  Fetching latest Firecracker version...
[info]  Downloading firecracker.tgz...
[ok]    Firecracker v1.14.2 installed
[info]  Downloading vmlinux-6.1...
[ok]    Kernel installed (vmlinux-6.1)
[info]  Downloading ubuntu-24.04.squashfs...
[info]  Converting squashfs to ext4 (this may take a minute)...
[ok]    Rootfs installed (ubuntu-24.04.ext4, 1024 MB)
[info]  Fetching latest release tag...
[ok]    Latest release: v0.1.0-alpha.24
[info]  Installing vmsan CLI via npm...
npm warn deprecated node-domexception@1.0.0: Use your platform's native DOMException instead

added 62 packages in 27s

8 packages are looking for funding
  run `npm fund` for details
npm notice
npm notice New major version of npm available! 10.9.4 -> 11.11.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.11.0
npm notice To update run: npm install -g npm@11.11.0
npm notice
[ok]    vmsan CLI installed (0.1.0-alpha.24)
[info]  Downloading vmsan-agent...
[ok]    vmsan-agent v0.1.0-alpha.24 installed
[info]  Downloading cloudflared...
[ok]    cloudflared 2026.2.0 installed

  ┌─────────────────────────────────────────────────────────────┐
  │  Cloudflare Tunnel (optional)                              │
  │                                                            │
  │  vmsan can expose VMs via Cloudflare Tunnels.              │
  │  You need a Cloudflare API token and a domain managed      │
  │  by Cloudflare.                                            │
  └─────────────────────────────────────────────────────────────┘

  Configure Cloudflare now? [y/N] N
[info]  Skipping Cloudflare configuration

[ok]    vmsan environment ready at /root/.vmsan

  Firecracker  /root/.vmsan/bin/firecracker
  Jailer       /root/.vmsan/bin/jailer
  Kernel       /root/.vmsan/kernels/vmlinux-6.1
  Rootfs       /root/.vmsan/rootfs/ubuntu-24.04.ext4
  Agent        /root/.vmsan/bin/vmsan-agent
  cloudflared  /root/.vmsan/bin/cloudflared (not configured)
  CLI          /usr/bin/vmsan

  To configure Cloudflare later, re-run this installer.

root@microvm:~#

Finally, we launch a microVM. We use the parameter --connect that gives as a non-root shell into the newly created microVM. The microVM has by default 128MiB RAM and a disk space of 10GiB. The Incus VM is also at 10GiB, which may not good. The default memory size is too little to even run apt get and the default disk space is too much. Don’t worry, we will remove this microVM and create a new one in the next section. What’s important, is that we got a shell into the microVM. It works!

root@microvm:~# vmsan create --connect
◐ Creating VM vm-918c59dc...                                                              
◐ Setting up networking...                                                                
✔ Network: TAP fhvm0, Host 172.16.0.1, Guest 172.16.0.2                                   
◐ Preparing chroot...                                                                     
◐ Spawning Firecracker via jailer...                                                      
◐ Waiting for API socket...                                                               
✔ API socket ready                                                                        
◐ Starting VM...                                                                          
✔ VM vm-918c59dc is running (PID: 2116)                                                   

 ╭───────────────────────────────────────────────────────────────────────────────────────╮
 │                                                                                       │
 │  VM Created: vm-918c59dc                                                              │
 │                                                                                       │
 │    Status:   running                                                                  │
 │    PID:      2116                                                                     │
 │    vCPUs:    1                                                                        │
 │    Memory:   128 MiB                                                                  │
 │    Runtime:  base                                                                     │
 │    Disk:     10 GB                                                                    │
 │                                                                                       │
 │    Network:                                                                           │
 │      TAP:    fhvm0                                                                    │
 │      Host:   172.16.0.1                                                               │
 │      Guest:  172.16.0.2                                                               │
 │      MAC:    AA:FC:00:00:00:01                                                        │
 │      Policy: allow-all                                                                │
 │                                                                                       │
 │    Kernel:   /root/.vmsan/kernels/vmlinux-6.1                                         │
 │    Rootfs:   /root/.vmsan/rootfs/ubuntu-24.04.ext4                                    │
 │                                                                                       │
 │    Socket:   /root/.vmsan/jailer/firecracker/vm-918c59dc/root/run/firecracker.socket  │
 │    Chroot:   /root/.vmsan/jailer/firecracker/vm-918c59dc                              │
 │    State:    /root/.vmsan/vms/vm-918c59dc.json                                        │
 │                                                                                       │
 ╰───────────────────────────────────────────────────────────────────────────────────────╯

◐ Waiting for agent to become ready...                                                    
✔ Agent is ready. Connecting via PTY shell...                                             
ubuntu@ubuntu-fc-uvm:~$

The microVM gets an IP address from the 176.16.x.y private address range. The hostname of the microVM is ubuntu-fc-uvm, and the name is derived from Ubuntu FireCracker microVirtualMachine.

Let’s list the available microVMs and then remove ’em. In the next section we examine a bit more about these commands.

root@microvm:~# vmsan list
ID            STATUS    CREATED         MEMORY    VCPUS  RUNTIME   TIMEOUT TUNNEL SNAPSHOT                                                                              
vm-918c59dc   running   5 minutes ago   128 MiB   1      base      -       -      -       
root@microvm:~# vmsan remove vm-63eade04

 ERROR  Cannot remove running VM(s): vm-63eade04. Stop them first or use --force (-f).

root@microvm:~# vmsan remove vm-63eade04 --force
◐ Removing vm-63eade04...
✔ Removed vm-63eade04
root@microvm:~# vmsan list
No VMs found.    
root@microvm:~#

Using `vmsan`

Here are the available commands for vmsan. We can create microVMs and we can list them. By listing them, we can see the name that was given to them so that we can start, stop and remove them. We can get a shell or run commands in a microVM with connect and exec. We transfer files with upload and download. Finally, with network we change the network policy.

root@microvm:~# vmsan 
Firecracker microVM sandbox toolkit (vmsan v0.1.0-alpha.24)

USAGE vmsan [OPTIONS] create|list|ls|start|stop|remove|rm|connect|upload|download|exec|network

OPTIONS

     --json    Output structured JSON (one event per command) 
  --verbose    Show detailed debug output with wide event tree

COMMANDS

    create    Create and start a Firecracker microVM                 
      list    List all VMs                                           
        ls    List all VMs                                           
     start    Start a previously stopped VM                          
      stop    Stop one or more running VMs                           
    remove    Remove one or more VMs (stops if running, then deletes)
        rm    Remove one or more VMs (stops if running, then deletes)
   connect    Connect to a running VM                                
    upload    Upload local files to a running VM                     
  download    Download a file from a running VM                      
      exec    Execute a command inside a running VM                  
   network    Update network policy on a running VM                  

Use vmsan  --help for more information about a command.

No command specified.
root@microvm:~#

`vmsan create` to create a microVM

Here are the parameters. Note that if you run without any parameters (like --help), it will plough ahead create a microVM with the default parameters.

By default, a microVM has 1 vCPU, 128MiB memory (you specify here the number only), disk space of 10gb (you specify here the number with gb). If you use the --connect parameter, you get a non-root shell once the microVM is created. The default image is base, which is currently an Ubuntu 24.04.3. Once you launch a microVM, the base image uses 388MB of disk space. In addition, the microVM from the base image occupies about 34MiB of memory.

root@microvm:~# vmsan create --help
Create and start a Firecracker microVM (vmsan create v0.1.0-alpha.24)

USAGE vmsan create [OPTIONS] 

OPTIONS

                   --vcpus="1"    Number of vCPUs (default: 1)                                                                                                                 
                --memory="128"    Memory in MiB (default: 128)                                                                                                                 
                      --kernel    Path to kernel image. Auto-detected from kernels/ if not specified.                                                                          
                      --rootfs    Path to rootfs image. Auto-detected from rootfs/ if not specified.                                                                           
                  --from-image    Build rootfs from a Docker/OCI image (e.g. ubuntu:latest).                                                                                   
              --runtime="base"    Runtime label (e.g. python3.13, node22, node22-demo). node22-demo auto-selects node:22 image and serves a branded welcome page. Default: base
                     --project    Project label for grouping VMs                                                                                                               
                 --disk="10gb"    Root disk size in GB (default: 10gb)                                                                                                         
                     --timeout    Auto-shutdown timeout (e.g. 1h, 30m, 2h30m)                                                                                                  
                --publish-port    Ports to forward to the VM (comma-separated, e.g. 8080,3000)                                                                                 
                    --snapshot    Snapshot ID to restore from                                                                                                                  
  --network-policy="allow-all"    Base network mode: allow-all (default), deny-all, or custom. Auto-promoted to custom when domains or CIDRs are provided.                     
              --allowed-domain    Domains/patterns to allow (comma-separated). Wildcard * for subdomains.                                                                      
                --allowed-cidr    Address ranges to allow (comma-separated CIDR, e.g. 10.0.0.0/8)                                                                              
                 --denied-cidr    Address ranges to deny (comma-separated CIDR). Takes precedence over all allows.                                                             
                  --no-seccomp    Disable seccomp-bpf filter for the Firecracker process.                                                                                      
                   --no-pid-ns    Disable PID namespace isolation for the jailer.                                                                                              
                   --no-cgroup    Disable cgroup resource limits for CPU and memory.                                                                                           
                    --no-netns    Disable per-VM network namespace isolation.                                                                                                  
                   --bandwidth    Max bandwidth per VM (e.g., 50mbit, 100mbit). Default: unlimited.                                                                            
                     --connect    Automatically connect to the VM shell after creation.                                                                                        
                      --silent    Suppress all output                                                                                                                          


root@microvm:~#

We are creating microVM with the following defaults.

root@microvm:~# vmsan create --disk="1gb" --memory="512" --connect
◐ Creating VM vm-4373c955...
◐ Setting up networking...
✔ Network: TAP fhvm0, Host 172.16.0.1, Guest 172.16.0.2 
◐ Preparing chroot...
◐ Spawning Firecracker via jailer... 
◐ Waiting for API socket...
✔ API socket ready 
◐ Starting VM...   
✔ VM vm-4373c955 is running (PID: 3987) 
 ╭───────────────────────────────────────────────────────────────────────────────────────╮
 │                                                                                       │
 │  VM Created: vm-4373c955                                                              │
 │                                                                                       │
 │    Status:   running                                                                  │
 │    PID:      3987                                                                     │
 │    vCPUs:    1                                                                        │
 │    Memory:   512 MiB                                                                  │
 │    Runtime:  base                                                                     │
 │    Disk:     1 GB                                                                     │
 │                                                                                       │
 │    Network:                                                                           │
 │      TAP:    fhvm0                                                                    │
 │      Host:   172.16.0.1                                                               │
 │      Guest:  172.16.0.2                                                               │
 │      MAC:    AA:FC:00:00:00:01                                                        │
 │      Policy: allow-all                                                                │
 │                                                                                       │
 │    Kernel:   /root/.vmsan/kernels/vmlinux-6.1                                         │
 │    Rootfs:   /root/.vmsan/rootfs/ubuntu-24.04.ext4                                    │
 │                                                                                       │
 │    Socket:   /root/.vmsan/jailer/firecracker/vm-4373c955/root/run/firecracker.socket  │
 │    Chroot:   /root/.vmsan/jailer/firecracker/vm-4373c955                              │
 │    State:    /root/.vmsan/vms/vm-4373c955.json                                        │
 │                                                                                       │
 ╰───────────────────────────────────────────────────────────────────────────────────────╯

◐ Waiting for agent to become ready...
✔ Agent is ready. Connecting via PTY shell...
ubuntu@ubuntu-fc-uvm:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.3 LTS"
ubuntu@ubuntu-fc-uvm:~$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       974M  388M  570M  41% /
ubuntu@ubuntu-fc-uvm:~$ free -h
               total        used        free      shared  buff/cache   available
Mem:           104Mi        34Mi        28Mi       1.7Mi        52Mi        70Mi
Swap:             0B          0B          0B
ubuntu@ubuntu-fc-uvm:~$ su
root@ubuntu-fc-uvm:/home/ubuntu# cd
root@ubuntu-fc-uvm:~# pwd
/root
root@ubuntu-fc-uvm:~# exit
exit
ubuntu@ubuntu-fc-uvm:~$ exit
root@microvm:~#

`vmsan list` to list the microVMs

We run vmsan list to list the available microVMs.

root@microvm:~# vmsan list
ID            STATUS    CREATED         MEMORY    VCPUS   RUNTIME TIMEOUT TUNNEL SNAPSHOT
vm-4373c955   running   7 minutes ago   512 MiB   1       base    -       -      -       
root@microvm:~#

`vmsan connect` to get a shell into a microVM

We run vmsan connect to get a shell into a microVM. We need the ID of the microVM, and we get it by running vmsan list. We can get root by running su. The root account does not have a password, therefore you are not asked for a password for the su command. sudo is not enabled in this image as it is not needed. I am running su --login in order to get a login shell for root.

root@microvm:~# vmsan connect vm-4373c955
ubuntu@ubuntu-fc-uvm:~$ su --login
root@ubuntu-fc-uvm:~#

Running nginx in a microVM

Let’s do something useful. We are going to run nginx in a microVM and we will access it from the host. We are going to start from scratch, creating a new microVM.

root@microvm:~# vmsan create --disk="1gb" --memory="512" --publish-port 80 --connect
◐ Creating VM vm-2f796b98...                                                              
◐ Setting up networking...                                                                
✔ Network: TAP fhvm0, Host 198.19.0.1, Guest 198.19.0.2                                   
◐ Preparing chroot...                                                                     
◐ Spawning Firecracker via jailer...                                                      
◐ Waiting for API socket...                                                               
✔ API socket ready                                                                        
◐ Starting VM...                                                                          
✔ VM vm-2f796b98 is running (PID: 6942)                                                   

 ╭───────────────────────────────────────────────────────────────────────────────────────╮
 │                                                                                       │
 │  VM Created: vm-2f796b98                                                              │
 │                                                                                       │
 │    Status:   running                                                                  │
 │    PID:      6942                                                                     │
 │    vCPUs:    1                                                                        │
 │    Memory:   512 MiB                                                                  │
 │    Runtime:  base                                                                     │
 │    Disk:     1 GB                                                                    │
 │                                                                                       │
 │    Network:                                                                           │
 │      TAP:    fhvm0                                                                    │
 │      Host:   198.19.0.1                                                               │
 │      Guest:  198.19.0.2                                                               │
 │      MAC:    AA:FC:00:00:00:01                                                        │
 │      Policy: allow-all                                                                │
 │      Ports:  80                                                                       │
 │                                                                                       │
 │    Kernel:   /root/.vmsan/kernels/vmlinux-6.1                                         │
 │    Rootfs:   /root/.vmsan/rootfs/ubuntu-24.04.ext4                                    │
 │                                                                                       │
 │    Socket:   /root/.vmsan/jailer/firecracker/vm-2f796b98/root/run/firecracker.socket  │
 │    Chroot:   /root/.vmsan/jailer/firecracker/vm-2f796b98                              │
 │    State:    /root/.vmsan/vms/vm-2f796b98.json                                        │
 │                                                                                       │
 ╰───────────────────────────────────────────────────────────────────────────────────────╯

◐ Waiting for agent to become ready...                                                    
✔ Agent is ready. Connecting via PTY shell...                                             
ubuntu@vm-2f796b98:~$

Note that if we try to run apt install, it will fail because it cannot find the packages. The reason is that this VM has not updated automatically the package list, and we have to perform this manually. lsof -i shows that the Web server has not started yet. We enable and then start the nginx service. We then verify that nginx is now running. Finally, we change the default index.html file to reflect that we are running the Web server in a microVM and all that in an Incus VM.

ubuntu@vm-2f796b98:~$ su -l
root@ubuntu-fc-uvm:~# 
root@ubuntu-fc-uvm:~# apt install nginx lsof
Reading package lists... Done
Building dependency tree... Done
E: Unable to locate package nginx
E: Unable to locate package lsof
root@ubuntu-fc-uvm:~# apt update
...
root@ubuntu-fc-uvm:~# apt install -y nginx lsof
...
root@ubuntu-fc-uvm:~# lsof -i
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd     1 root   64u  IPv4   1507      0t0  TCP *:ssh (LISTEN)
systemd     1 root   65u  IPv6   1511      0t0  TCP *:ssh (LISTEN)
vmsan-age 582 root    3u  IPv6   1630      0t0  TCP *:9119 (LISTEN)
vmsan-age 582 root    8u  IPv6   1953      0t0  TCP 172.16.0.2:9119->10.200.0.1:39920 (ESTABLISHED)
root@vm-f1d2f963:~# systemctl enable nginx
Synchronizing state of nginx.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable nginx
root@ubuntu-fc-uvm:~# systemctl start nginx
root@ubuntu-fc-uvm:~# lsof -i
COMMAND    PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd      1     root   64u  IPv4   1507      0t0  TCP *:ssh (LISTEN)
systemd      1     root   65u  IPv6   1511      0t0  TCP *:ssh (LISTEN)
vmsan-age  582     root    3u  IPv6   1630      0t0  TCP *:9119 (LISTEN)
vmsan-age  582     root    8u  IPv6   1953      0t0  TCP 172.16.0.2:9119->10.200.0.1:39920 (ESTABLISHED)
nginx     6342     root    5u  IPv4  11303      0t0  TCP *:http (LISTEN)
nginx     6342     root    6u  IPv6  11304      0t0  TCP *:http (LISTEN)
nginx     6343 www-data    5u  IPv4  11303      0t0  TCP *:http (LISTEN)
nginx     6343 www-data    6u  IPv6  11304      0t0  TCP *:http (LISTEN)
root@vm-f1d2f963:~# sed -i "s/Welcome to nginx/Welcome to nginx running in a vmsan microMV, which runs in an Incus VM/g"  /var/www/html/index.nginx-debian.html 
root@ubuntu-fc-uvm:~#

We exit to the host and create the Incus proxy device. While exiting, we have a look at the firewall rule that performed the publishing of port 80 (http) to the Incus virtual machine.

root@ubuntu-fc-uvm:~# exit
exit
ubuntu@ubuntu-fc-uvm:~$ exit
exit
root@microvm:~# iptables-nft 
...
Chain FORWARD (policy DROP)
ACCEPT     tcp  --  anywhere             198.19.0.2           tcp dpt:http
...
root@microvm:~# exit
logout
$

Here is a screenshot of the the website, when accessed from the host. We use the IP address of the Incus VM microvm.

Conclusion

We had a quick look into vmsan, a project that manages micro virtual machines that are based on Firecracker. vmsan is a new project and the maintainer is Angelo RC.

vmsan also added support for Cloudflare tunnels. This means that you can expose your microVMs to the Internet using those Cloudflare tunnels. If you own a domain and you have configured it to be managed by Cloudflare, then vmsan allows you to publish your site automatically without needing to enable port forwarding. You would need to create an API key for this on Cloudflare and then feed that API key to vmsan.

At the moment vmsan comes with four images.

Ubuntu 24.04
Ubuntu 24.04 with Python 3.13
Ubuntu 24.04 with Node.js 22 LTS
Ubuntu 24.04 with Node.js 24 LTS

I would expect that the project will add support for smaller images, such as Alpine. Considering that Firecracker is not QEMU and has fewer requirements for resources (and features!), it would make sense to have tiny VMs.

vmsan itself is a Node.js application.

How to run a Linux Desktop virtual machine on Incus

Simos Xenitellis — Thu, 01 May 2025 22:51:53 +0000

Incus is a manager for virtual machines and system containers.

A virtual machine (VM) is an instance of an operating system that runs on a computer, along with the main operating system. A virtual machine uses hardware virtualization features for the separation from the main operating system. With virtual machines, the full operating system boots up in them. While in most cases you would run Linux on a VM without a desktop environment, you can also run Linux with a desktop environment (like in VirtualBox and VMWare).

In How to run a Windows virtual machine on Incus on Linux we saw how to run a run a Windows VM on Incus. In this post we see how to run a Linux Desktop virtual machine on Incus.

Updates
Prerequisites
Cheat sheet
Availability of images
Booting a desktop Linux VM on Incus
Troubleshooting

Updates

No updates yet.

Prerequisites

You should have a system that runs Incus.
A system with support for hardware virtualization so that it can run virtual machines.
A virtual machine image of your preferred Linux desktop distribution.

Cheat sheet

You should specify how much RAM memory you are giving to the VM. The default is only 1GiB of RAM, which is not enough for desktop VMs. The --console=vga launches for you the Remote Viewer GUI application to allow you to use the desktop in a window.

$ incus image list images:desktop       # List all available desktop images
$ incus launch --vm images:ubuntu/jammy/desktop mydesktop -c limits.memory=3GiB --console=vga
$ incus console mydesktop --type=vga    # Reconnect to already running instance
$ incus start mydesktop --console=vga   # Start an existing desktop VM

Availability of images

Currently, Incus provides you with the following VM images of Linux desktop distributions. The architecture is x86_64.

Run the following command to list all available Linux desktop images. incus image is the section of Incus that deals with the management of images. The list command lists the available images of a remote/repository, the default being images: (run incus remote list for the full list of remotes). After the colon (:), you type filter keywords, and in this case we typed desktop to show images that have the word desktop in them (to show only Desktop images). We are interested in a few columns only, therefore -c ldt only shows the columns for the Alias, the Description and the Type.

$ incus image list images:desktop -c ldt
+------------------------------------------+---------------------------+-----------------+
|                  ALIAS                   |      DESCRIPTION          |      TYPE       |
+------------------------------------------+---------------------------+-----------------+
| archlinux/desktop-gnome (3 more)         | Archlinux current amd64   | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
| opensuse/15.5/desktop-kde (1 more)       | Opensuse 15.5 amd64       | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
| opensuse/15.6/desktop-kde (1 more)       | Opensuse 15.6 amd64       | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
| opensuse/tumbleweed/desktop-kde (1 more) | Opensuse tumbleweed amd64 | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
| ubuntu/24.10/desktop (3 more)            | Ubuntu oracular amd64     | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
| ubuntu/focal/desktop (3 more)            | Ubuntu focal amd64        | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
| ubuntu/jammy/desktop (3 more)            | Ubuntu jammy amd64        | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
| ubuntu/noble/desktop (3 more)            | Ubuntu noble amd64        | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
| ubuntu/plucky/desktop (1 more)           | Ubuntu plucky amd64       | VIRTUAL-MACHINE |
+------------------------------------------+---------------------------+-----------------+
$

These images have been generated with the utility distrobuilder, https://github.com/lxc/distrobuilder The purpose of the utility is to prepare the images so that when we launch them, we get immediately the desktop environment and do not perform any manual configuration. The configuration files for distrobuilder to create these images can be found at https://github.com/lxc/lxc-ci/tree/main/images For example, the archlinux.yaml configuration file has a section to create the desktop image, along with the container and other virtual machine images.

The full list of Incus images are also available on the Web, through the website https://images.linuxcontainers.org/ It is possible to generate more such desktop images by following the steps of the existing configuration files. Perhaps a Kali Linux desktop image would be very useful. In the https://images.linuxcontainers.org/ website you can also view the build logs that were generated while building the images, and figure out what parameters are needed for distrobuilder to build them (along with the actual configuration file). For example, here are the logs for the ArchLinux desktop image, https://images.linuxcontainers.org/images/archlinux/current/amd64/desktop-gnome/

Up to this point we got a list of the available virtual machine images that are provided by Incus. We are ready to boot them.

Booting a desktop Linux VM on Incus

When launching a VM, Incus provides by default 1GiB RAM and 10GiB of disk space. The disk space is generally OK, but the RAM is too little for a desktop image (it’s OK for non-desktop images). For example, for an Ubuntu desktop image, the instance requires about 1.2GB of memory to start up and obviously more to run other programs. Therefore, if we do not specify more RAM, then the VM would struggle to make do of the mere 1GiB of RAM.

Booting the Ubuntu desktop image on Incus

Here is the command to launch a desktop image. We use incus launch to launch the image. It’s a VM, hence --vm. We are using the image from the images: remote, the one called ubuntu/plucky/desktop (it’s the last from the list of the previous section). We configure a new limit for the memory usage, -c limits.memory=3GiB, so that the instance will be able to run successfully. Finally, the console is not textual but graphical. We specify that with --console=vga which means that Incus will launch the remote desktop utility for us.

$ incus launch --vm images:ubuntu/plucky/desktop mydesktop -c limits.memory=3GiB --console=vga
Launching mydesktop

Here is a screenshot of the new window with the running desktop virtual machine.

Screenshot of images:ubuntu/plucky/desktop

Now we closed the wizard.

Screenshot of images:ubuntu/plucky/desktop after we close the wizard.

Booting the ArchLinux desktop image on Incus

I cannot get this image to show the desktop. If someone can make this work, please post in a comment.

$ incus launch --vm images:archlinux/desktop-gnome mydesktop -c limits.memory=3GiB --console=vga -c security.secureboot=false
Launching mydesktop

Booting the OpenSUSE desktop image on Incus

$ incus launch --vm images:opensuse/15.5/desktop-kde mydesktop -c limits.memory=3GiB --console=vga
Launching mydesktop

Troubleshooting

I closed the desktop window but the VM is running. How do I get it back up?

If you closed the Remote Viewer window, you can get Incus to start it again with the following command. By doing so, you are actually reconnecting back to the VM and continue working from where you left off.

We are using the incus console action to connect to the running mydesktop instance and request access through the Remote Viewer (rather than a text console).

$ incus console mydesktop --type=vga

Error: This console is already connected. Force is required to take it over.

You are already connected to the desktop VM with the Remote Viewer and you are trying to connect again. Either go to the existing Remote Viewer window, or add the parameter --force to close the existing Remote Viewer window and open a new one.

Error: Instance is not running

You are trying to connect to a desktop VM with the Remote Viewer but the instance (which already exists) is not running. Use the action incus start to start the virtual machine, along with the --type=vga parameter to get Incus to launch the Remote Viewer for you.

$ incus start mydesktop --console=vga

I get no audio from the desktop VM! How do I get sound in the desktop VM?

This requires extra steps which I do not show yet. There are three options. The first is to use the QEMU device emulation to emulate a sound device in the VM. The second is to somehow push an audio device into the VM so that this audio device is used exclusively in the VM (have not tried this but I think it’s possible). The third and perhaps best option is to use network audio with PulseAudio/Pipewire. You enable network audio on your desktop and then configure the VM instance to connect to that network audio server. I have tried that and it worked well for me. The downside is that the Firefox snap package in the VM could not figure out that there is network audio there and I could not get audio in that application.

How do I shutdown the desktop VM?

Use the desktop UI to perform the shutdown. The VM will shut down cleanly.

Error: Failed instance creation: The image used by this instance is incompatible with secureboot. Please set security.secureboot=false on the instance

You tried to launch a virtual machine with SecureBoot enabled but the image does not support SecureBoot. You need to disable SecureBoot when you launch this image. The instance has been created but is unable to run unless you disable SecureBoot. You can either disable SecureBoot through an Incus configuration for this image, or just delete the instance, and try again with the parameter -c security.secureboot=false.

Here is how to disable SecureBoot, then try to incus start that instance.

$ incus config set mydesktop security.secureboot=true

Here is how you would enable that flag when you launch such a VM.

incus launch --vm images:archlinux/desktop-gnome mydesktop -c limits.memory=3GiB --console=vga -c security.secureboot=false

Note that official Ubuntu images can work with SecureBoot enabled, most others don’t. It has to do with the Linux kernel being digitally signed by some certification authority.

Error: Failed instance creation: Failed creating instance record: Add instance info to the database: Failed to create “instances” entry: UNIQUE constraint failed: instances.project_id, instances.name

This error message is a bit cryptic. It just means that you are trying to create or launch an instance while the instance already exists. Read as Error: The instance name already exists.

A networking guide for Incus

Simos Xenitellis — Sun, 29 Sep 2024 14:12:22 +0000

Incus is a hypervisor/manager for virtual machines and application/system containers. Get community support here.

A system container is an instance of an operating system that also runs on a computer, along with the main operating system. A system container, instead, uses security primitives of the Linux kernel for the separation from the main operating system. The system container follows the lifecycle of a computer system. You can think of system containers as software virtual machines.

An application container is a container that has an application or service. It follows the lifecycle of the application instead of a system. That is, here you start and stop the application instead of booting and shutting down a system. Incus supports Open Container Initiative (OCI) images such as Docker images. When Incus launches an OCI image, it uses its own runtime, not Docker’s. That is, Incus consumes images from any OCI image repositories.

In virtual machines and system/application containers we can attach virtual networking devices, either

none, (i.e. an instance without networking)
one or, (i.e. most common and simple case)
more than one.

In addition to the virtual networking devices, we can also attach real hardware networking devices. Those devices can be taken away from the host and get pushed into a virtual machine or system container.

You may use a combination of those networking devices in the same instance. It is left as an exercise to the reader to explore that road. In these tutorials we are look at one at most networking device per instance.

There will be attempts to generalize and explain in practical terms. If I get something wrong, please correct me in the comments so that it gets fixed and we all learn something new. Note that I will be editing this content along the way, adding material, troubleshooting cases, etc.

In this post we are listing tutorials of the different Incus devices of type nic (network interface controller). Whatever we write in this post and the linked tutorials, are covered in that documentation URL!

The list of tutorials per networking:

bridge (the default, the local network bridge), it’s in this post below.
bridged, (pending)
macvlan, (pending)
none,
physical,
ipvlan,
routed,

The setup

When demonstrating these network configurations, we will be using an Incus VM. When learning, try there in your Incus VM before applying on your host or your server.

We launch an Incus VM, called tutorial, with Ubuntu 24.04 LTS, then get a shell with the default non-root account ubuntu. I am impatient and I am typing repeatedly the incus exec command to get a shell. The VM takes a few moments to boot up, and I get interested error messages until the VM is actually running. Not really relevant to this tutorial but you will get educated at every opportunity.

$ incus launch images:ubuntu/24.04/cloud tutorial --vm
Launching tutorial
$ incus exec tutorial -- su -l ubuntu
Error: VM agent isn't currently running
$ incus exec tutorial -- su -l ubuntu
su: user ubuntu does not exist or the user entry does not contain all the required fields
$ incus exec tutorial -- su -l ubuntu
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@tutorial:~$

We got a shell in the VM. Then, install Incus which is available in the default repositories of Ubuntu 24.04 LTS. Also, we install zfsutils-linux, which are the client utilities to use ZFS in Incus. We are advised to add our non-root account to the incus-admin group in order to have access to Incus. Without that, we would have to use sudo all the time. When you add a user to a group, you need to logout then log in again for the change to take effect. And this is what we do (unless you know about newgrp).

ubuntu@tutorial:~$ sudo apt install -y incus zfsutils-linux
...
Creating group 'incus' with GID 989.
Creating group 'incus-admin' with GID 988.
Created symlink /etc/systemd/system/multi-user.target.wants/incus-startup.service → /usr/lib/systemd/system/incus-startup.service.
Created symlink /etc/systemd/system/sockets.target.wants/incus-user.socket → /usr/lib/systemd/system/incus-user.socket.
Created symlink /etc/systemd/system/sockets.target.wants/incus.socket → /usr/lib/systemd/system/incus.socket.
incus.service is a disabled or a static unit, not starting it.
incus-user.service is a disabled or a static unit, not starting it.

Incus has been installed. You must run `sudo incus admin init` to
perform the initial configuration of Incus.
Be sure to add user(s) to either the 'incus-admin' group for full
administrative access or the 'incus' group for restricted access,
then have them logout and back in to properly setup their access.

...
ubuntu@tutorial:~$ sudo usermod -a -G incus-admin ubuntu
ubuntu@tutorial:~$ logout
$ incus exec tutorial -- su -l ubuntu
ubuntu@tutorial:~$

Now we initialize Incus with sudo incus admin init.

Default Incus networking

When you install and setup Incus with incus admin init, you are prompted whether you want to create a local network bridge. We press Enter to all prompts, which means that we accept all the defaults that are presented to us. The last question is whether to show the initialization configuration. If you missed it, you can get it after the fact by running incus admin init --dump (dumps the configuration).

ubuntu@tutorial:~$ incus admin init
Would you like to use clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (zfs, dir) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=incusbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks:
- config:
    ipv4.address: auto
    ipv6.address: auto
  description: ""
  name: incusbr0
  type: ""
  project: default
storage_pools:
- config:
    size: 5GiB
  description: ""
  name: default
  driver: zfs
profiles:
- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      network: incusbr0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null

ubuntu@tutorial:~$

If you accept the defaults (i.e. press Enter in each) or type them explicitly, you get a local bridge named incusbr0 that is managed by Incus, and gives private IPv4 and IPv6 IP addresses to your newly created instances.

Let’s see them in practice in your Incus installation. You have configured Incus and Incus created a default profile, called default, for you. This profile is applied by default to all newly created instances and has the networking configuration in there. In that profile there are two devices, and one of them is the networking device. In Incus the device is called eth0 (in pink color), and in the instance it will be shown as eth0 (green color). On the host, the bridge will appear with the name incusbr0. It’s a networking type, hence of type nic.

ubuntu@tutorial:~$ incus profile list
+---------+-----------------------+---------+
|  NAME   |      DESCRIPTION      | USED BY |
+---------+-----------------------+---------+
| default | Default Incus profile | 0       |
+---------+-----------------------+---------+
ubuntu@tutorial:~$ incus profile show default
config: {}
description: Default Incus profile
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by: []
ubuntu@tutorial:~$

incusbr0 was created by Incus. Let’s see details through the incus network commands. We first list the network interfaces and then we show the incusbr0 network interface. incusbr0 is a managed network interface (in pink below), and it’s managed by Incus. Incus takes care of the networking and provides DHCP services, and access to the upstream network (i.e. the Internet). incusbr0 is a network bridge (in blue). An instance that requires network configuration from incusbr0, will get an IP address from the range 10.180.234.1-254 (in orange). Network Address Translation (NAT) is enabled (also in orange), which means there is access to the upstream network, and likely the Internet.

ubuntu@tutorial:~$ incus network list
+----------+----------+---------+-----------------+---------+---------+
|   NAME   |   TYPE   | MANAGED |      IPV4       | USED BY |  STATE  |    
+----------+----------+---------+-----------------+---------+---------+
| enp5s0   | physical | NO      |                 | 0       |         |    
+----------+----------+---------+-----------------+---------+---------+
| incusbr0 | bridge   | YES     | 10.180.234.1/24 | 1       | CREATED |
+----------+----------+---------+-----------------+---------+---------+
ubuntu@tutorial:~$ incus network show incusbr0
config:
  ipv4.address: 10.180.234.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:7:7dfe:75cf::1/64
  ipv6.nat: "true"
description: ""
name: incusbr0
type: bridge
used_by:
- /1.0/profiles/default
managed: true
status: Created
locations:
- none
ubuntu@tutorial:~$

Let’s launch a container and test these out. The instance got an IP address (in orange) that is within the range of the network bridge above.

ubuntu@tutorial:~$ incus launch images:alpine/edge/cloud myalpine
Launching myalpine
ubuntu@tutorial:~$ incus list -c ns4t     
+----------+---------+----------------------+-----------+
|   NAME   |  STATE  |         IPV4         |   TYPE    |
+----------+---------+----------------------+-----------+
| myalpine | RUNNING | 10.180.234.24 (eth0) | CONTAINER |
+----------+---------+----------------------+-----------+
ubuntu@tutorial:~$

The IP address is OK but could it look better? It’s private anyway, and we can select anything from the range 10.x.y.z. Let’s change it so that it uses instead 10.10.10.1-254. We set the configuration of incusbr0 for ipv4.address (see earlier) to a new value, 10.10.10.1/24. Each number separated by commas is 8 bits in length, and /24 means that the first 3 * 8 = 24 bits should stay the same. We make the change, but the instance still has the old IP address. We restart the instance, and it automatically gets the new IP address from the new range.

ubuntu@tutorial:~$ incus network set incusbr0 ipv4.address=10.10.10.1/24
ubuntu@tutorial:~$ incus list -c ns4t
+----------+---------+----------------------+-----------+
|   NAME   |  STATE  |         IPV4         |   TYPE    |
+----------+---------+----------------------+-----------+
| myalpine | RUNNING | 10.180.234.24 (eth0) | CONTAINER |
+----------+---------+----------------------+-----------+
ubuntu@tutorial:~$ incus restart myalpine
ubuntu@tutorial:~$ incus list -c ns4t
+----------+---------+--------------------+-----------+
|   NAME   |  STATE  |        IPV4        |   TYPE    |
+----------+---------+--------------------+-----------+
| myalpine | RUNNING | 10.10.10.24 (eth0) | CONTAINER |
+----------+---------+--------------------+-----------+
ubuntu@tutorial:~$

We have created incusbr0. Are we allowed to create another private bridge? Sure we are. We will call it incusbr1, and also we disable IPv6 networking. IPv6 addresses are too wide and mess up the formatting on my blog. If you notice earlier, there were no IPv6 addresses although IPv6 was configured on incusbr0. I cheated and removed the IPv6 addresses in some command outputs.

ubuntu@tutorial:~$ incus network create incusbr1 ipv4.address=10.10.20.1/24 ipv6.address=none
Network incusbr1 created
ubuntu@tutorial:~$ incus network show incusbr1
config:
  ipv4.address: 10.10.20.1/24
  ipv6.address: none
description: ""
name: incusbr1
type: bridge
used_by: []
managed: true
status: Created
locations:
- none
ubuntu@tutorial:~$

We have created incusbr1. Can we now launch an instance onto that private bridge? We launch the instance called myalpine1 and we used the incus launch parameter --network incusbr1 to specify a different network than the default network in the default Incus profile. We verify below that myalpine1 is served by incusbr1 (in green).

ubuntu@tutorial:~$ incus launch images:alpine/edge/cloud myalpine1 --network incusbr1
Launching myalpine1
ubuntu@tutorial:~$ incus list -c ns4t
+-----------+---------+--------------------+-----------+
|   NAME    |  STATE  |        IPV4        |   TYPE    |
+-----------+---------+--------------------+-----------+
| myalpine  | RUNNING | 10.10.10.24 (eth0) | CONTAINER |
+-----------+---------+--------------------+-----------+
| myalpine1 | RUNNING | 10.10.20.85 (eth0) | CONTAINER |
+-----------+---------+--------------------+-----------+
ubuntu@tutorial:~$ incus network show incusbr1
config:
  ipv4.address: 10.10.20.1/24
  ipv6.address: none
description: ""
name: incusbr1
type: bridge
used_by:
- /1.0/instances/myalpine1
managed: true
status: Created
locations:
- none
ubuntu@tutorial:~$

Technical details

The instances that use the Incus private bridge have access to the Internet. How is this achieved? It’s achieve with either iptables or nftables rules. In recent versions of Linux distributions, you would be using nftables by default (command: nft, no relation to NFTs). To view the firewall ruleset that were created by Incus, run sudo nft list ruleset. Here is my ruleset and should be similar to yours. There is one table for Incus and four chains. A persistent, a forward, an in and an out. More at the documentation site at nftables.

ubuntu@tutorial:~$ sudo nft list ruleset
table inet incus {
	chain pstrt.incusbr0 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 10.57.39.0/24 ip daddr != 10.57.39.0/24 masquerade
		ip6 saddr fd42:e7b:739c:7117::/64 ip6 daddr != fd42:e7b:739c:7117::/64 masquerade
	}

	chain fwd.incusbr0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "incusbr0" accept
		ip version 4 iifname "incusbr0" accept
		ip6 version 6 oifname "incusbr0" accept
		ip6 version 6 iifname "incusbr0" accept
	}

	chain in.incusbr0 {
		type filter hook input priority filter; policy accept;
		iifname "incusbr0" tcp dport 53 accept
		iifname "incusbr0" udp dport 53 accept
		iifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		iifname "incusbr0" udp dport 67 accept
		iifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
		iifname "incusbr0" udp dport 547 accept
	}

	chain out.incusbr0 {
		type filter hook output priority filter; policy accept;
		oifname "incusbr0" tcp sport 53 accept
		oifname "incusbr0" udp sport 53 accept
		oifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		oifname "incusbr0" udp sport 67 accept
		oifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
		oifname "incusbr0" udp sport 547 accept
	}
}
ubuntu@tutorial:~$

Future considerations

Network isolation.

How to recover or reconnect an Incus storage pool

Simos Xenitellis — Wed, 11 Sep 2024 14:05:17 +0000

Incus is a manager for virtual machines, system containers and application containers. Get Incus support here.

When you initially setup Incus, you create a storage pool where Incus will put in there everything. There are several options for storage pools, in this post we focus on ZFS storage pools, and those specifically that are stored on a separate block device (like /dev/sdb).

We are dealing with two cases. One, your installation of Incus has been somehow removed but the storage pool is somewhere there intact and you want to recover by installing again Incus. Two, you want to move the disk with storage pool from one computer to another, like reconnecting the storage pool on a new server.

This type of task is quite risky if you have a lot of important data on your system. Obviously, prior to you actually doing this on an actual system, you should take backups with incus export of your most important instances. And then, you should perform this tutorial several times so that you get the gist of recovering Incus installations. This tutorial shows you how to do a dry run of creating an Incus installation, killing it off, and then miraculously recovering it.

Prerequisites

You should have a running Incus installation.

Setting up Incus, using a block storage volume

We launch an Incus virtual machine (VM) that will act as our Incus server. We then (on the host) create a storage volume of type block. Next, we attach that block storage volume to the VM. In the VM it can be found as /dev/sdb. Subsequently, we incus admin init to initialize Incus, and configure Incus to use the block device /dev/sdb when creating the storage pool. When we run incus admin init, we press Enter when we want to accept the default value.

$ incus launch images:ubuntu/24.04/cloud --vm incusserver
Launching incusserver
$ incus storage volume create default IncusStorage --type=block size=6GiB
Storage volume IncusStorage created
$ incus storage volume attach default IncusStorage incusserver
$ incus shell incusserver
root@incusserver:~# fdisk -l /dev/sdb
Disk /dev/sdb: 6 GiB, 6442450944 bytes, 12582912 sectors
Disk model: QEMU HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@incusserver:~# sudo apt install -y incus zfsutils-linux
...
root@incusserver:~# incus admin init
Would you like to use clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (dir, zfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: yes
Path to the existing block device: /dev/sdb
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=incusbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks:
- config:
    ipv4.address: auto
    ipv6.address: auto
  description: ""
  name: incusbr0
  type: ""
  project: default
storage_pools:
- config:
    source: /dev/sdb
  description: ""
  name: default
  driver: zfs
profiles:
- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      network: incusbr0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null

root@incusserver:~#

Next we populate the Incus installation with a few alpines. We do this because we want to see these containers again after we recover the storage pool.

root@incusserver:~# incus launch images:alpine/edge alpine1
Launching alpine1
root@incusserver:~# incus launch images:alpine/edge alpine2
Launching alpine2
root@incusserver:~# incus launch images:alpine/edge alpine3
Launching alpine3
root@incusserver:~#

This is where the interesting stuff start. We now want to shutdown the Incus server and remove it. However, the block storage volume will still be there and in good condition, as the server has been shutdown cleanly. Note that the block storage volumes should only be attached to one system at a time.

root@incusserver:~# shutdown -h now
root@incusserver:~# Error: websocket: close 1006 (abnormal closure): unexpected EOF
$ incus storage volume show default IncusStorage
config:
  size: 6GiB
description: ""
name: IncusStorage
type: custom
used_by:
- /1.0/instances/incusserver
location: none
content_type: block
project: default
created_at: ...
$ incus delete incusserver
$ incus storage volume show default IncusStorage
config:
  size: 6GiB
description: ""
name: IncusStorage
type: custom
used_by: []
location: none
content_type: block
project: default
created_at: ...
$

Next, we launch a new VM that will be used as a new Incus server, then attach back the block storage volume with incus storage volume attach and install Incus along with the necessary ZFS client utils.

$ incus launch images:ubuntu/24.04/cloud --vm incusserver
Launching incusserver
$ incus storage volume attach default IncusStorage incusserver
$ incus shell incusserver
Error: Instance is not running
$ incus shell incusserver
Error: VM agent isn't currently running
$ incus shell incusserver
Error: VM agent isn't currently running
$ incus shell incusserver
Error: VM agent isn't currently running
$ incus shell incusserver
Error: VM agent isn't currently running
$ incus shell incusserver
Error: VM agent isn't currently running
$ incus shell incusserver
Error: VM agent isn't currently running
$ incus shell incusserver
root@incusserver:~# apt install -y zfsutils-linux incus
...
root@incusserver:~#

Finally, we bring back the old installation data with those three alpines. We run zpool import, which is a ZFS command that will look for potential ZFS pools and list them by name. The command zpool import default is the one that does the actual import. The ZFS pool name default was the name that was given by Incus before, when we were initializing Incus. Subsequently, we run incus admin recover to recover the ZFS pool and reconnect it with this new installation of Incus.

root@incusserver:~# zfs list
no datasets available
root@incusserver:~# zpool list
no pools available
root@incusserver:~# zpool import
   pool: default
     id: 8311839500301555365
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

	default     ONLINE
	  sdb       ONLINE
root@incusserver:~# zpool import default
root@incusserver:~# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
default  5.50G  6.80M  5.49G        -         -     0%     0%  1.00x    ONLINE  -
root@incusserver:~# 
root@incusserver:~# incus admin recover
This server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: yes
Name of the storage pool: default
Name of the storage backend (zfs, dir): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): /dev/sdb
Additional storage pool configuration property (KEY=VALUE, empty when done): 
Would you like to recover another storage pool? (yes/no) [default=no]: 
The recovery process will be scanning the following storage pools:
 - NEW: "default" (backend="zfs", source="/dev/sdb")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
The following unknown storage pools have been found:
 - Storage pool "default" of type "zfs"
The following unknown volumes have been found:
 - Container "alpine2" on pool "default" in project "default" (includes 0 snapshots)
 - Container "alpine3" on pool "default" in project "default" (includes 0 snapshots)
 - Container "alpine1" on pool "default" in project "default" (includes 0 snapshots)
Would you like those to be recovered? (yes/no) [default=no]: yes
Starting recovery...
root@incusserver:~# incus list
+---------+---------+------+------+-----------+-----------+
|  NAME   |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+---------+---------+------+------+-----------+-----------+
| alpine1 | STOPPED |      |      | CONTAINER | 0         |
+---------+---------+------+------+-----------+-----------+
| alpine2 | STOPPED |      |      | CONTAINER | 0         |
+---------+---------+------+------+-----------+-----------+
| alpine3 | STOPPED |      |      | CONTAINER | 0         |
+---------+---------+------+------+-----------+-----------+
root@incusserver:~#

Those alpines are in a STOPPED state. Will they start? Sure they will.

root@incusserver:~# incus start alpine1 alpine2 alpine3
root@incusserver:~# incus list -c ns4t
+---------+---------+----------------------+-----------+
|  NAME   |  STATE  |         IPV4         |   TYPE    |
+---------+---------+----------------------+-----------+
| alpine1 | RUNNING | 10.36.146.69 (eth0)  | CONTAINER |
+---------+---------+----------------------+-----------+
| alpine2 | RUNNING | 10.36.146.101 (eth0) | CONTAINER |
+---------+---------+----------------------+-----------+
| alpine3 | RUNNING | 10.36.146.248 (eth0) | CONTAINER |
+---------+---------+----------------------+-----------+
root@incusserver:~#

In this tutorial we saw how to recover an Incus installation, while the storage pool is intact. We covered the case that the storage pool is ZFS on a block device.

How to use your Android phone with ADB in an Incus container

Simos Xenitellis — Thu, 22 Aug 2024 15:35:15 +0000

Incus is a manager for virtual machines and system containers.

A system container (therein container) is an instance of an operating system that also runs on a computer, along with the main operating system. A system container uses, instead, security primitives of the Linux kernel for the separation from the main operating system. You can think of system containers as software virtual machines.

In this post we are going to see how to conveniently let an Incus container to have access to the Android USB debugging (ADB) of our Android mobile phone. Normally, you would run adb commands on the host. But in this case, we are launching a container and give it access to ADB. We set it up so that only the container can access the phone with ADB.

The reason why I am writing this post is that there are several pitfalls along the way. Let’s figure out how it all works.

Setting up the Android phone for USB Debugging

To enable USB debugging on your Android phone, there’s a hidden list of steps that involves going into your phone settings, tapping seven times on the appropriate place of the About page, and then your phone shows the message You are now a developer. You will need to search for the exact steps for your device, as the place where you need to tap seven times may be different among manufacturers.

Still on the phone, you then need to visit another location in the phone Settings, one that is called Developer options. In there, you need to scroll a bit until you see the option USB debugging. You will need to enable that. When you click to enable, you will be presented with a warning dialog box about the ramifications of having enabled the USB debugging. Read that carefully, and enable USB debugging. Note that after this exercise, and if you do not need to use ADB for a long period, you should disable the Developer options. The risk with the enabled USB debugging is that if you connect your phone with a (data) USB cable to some malicious computer or even a malicious USB charger, they may take over your device in a very bad way.

In the first screenshot it shows the warning when you try to enable USB debugging. The second screenshot shows that the USB debugging has been enabled successfully.

There’s an option in the second screenshot to Revoke USB debugging authorizations. I recommend to do that, especially if you have already connected your phone to your computer. By doing so, we can make sure that the host is not able to connect successfully to the device, and only the container can do so. Note that when you try to connect to the device with adb, you get a dialog box on the phone on whether to authorize this new connection.

When you connect your Android phone to your Linux host, the device should appear in the output of lsusb (list USB devices) as follows. I think the USB Vendor and Product IDs should be the same, 0x18d1 and 0x4ee7 respectively. And it should say at the end “(charging + debug)“. If you get something else, then you fell for the Android notification that says Use USB for [File Transfer / Android Auto]. That’s not good for us that we need USB debugging. The proper setting is Use USB for [No data transfer]. Yeah, a bit counter-intuitive.

$ lsusb
...
Bus 005 Device 005: ID 18d1:4ee7 Google Inc. Nexus/Pixel Device (charging + debug)
...
$

Setting up the host

In order to let the container have access to the phone, we need to make sure that there is no adb command on the host that is running. We can make sure that this is the case, if we run the following. If the ADB server is running on the host, then the container does not have access to the device. Interestingly, if you setup the container properly and ADB is running on the host, then as soon as you adb kill-server on the host, the container immediately has access to the device.

sudo adb kill-server

Creating the container

We are creating the container, we will called it adb, and it has access to USB debugging on the Android phone. The way we work with Incus is that we create a container with the task of accessing the phone, and then keep that container whenever we need to access the phone. In the incus config device command, we add to the adb container a device called adb(can use any name here), which is of type usb, and has the vendor and product IDs shown below. Finally, we restart the container.

$ incus launch images:debian/12/cloud adb
Launching adb
$ incus config device add adb adb usb vendorid=18d1 productid=4ee7
Device adb added to adb
user@user-desktop:~$ incus exec adb -- apt install -y adb
...
$ incus restart adb
$

Using adb in the Incus container

Let’s run adb devices in the container. You will most likely get unauthorized. When you run this command, your phone will prompt you whether you want to authorize this access. You should select to authorize the access.

$ incus exec adb -- adb devices
* daemon not running; starting now at tcp:5037
* daemon started successfully
List of devices attached
42282370	unauthorized
$

Now, run again the command.

$ incus exec adb -- adb devices
List of devices attached
42282370	device
$

And that’s it.

Stress testing adb in the Incus container

You would like to make this setup as robust as possible. One step is to remove the adb binary from the host.

Another test is to restart the container, and then check whether it still has access to the device.

$ incus restart adb
$ incus exec adb -- adb devices
* daemon not running; starting now at tcp:5037
* daemon started successfully
List of devices attached
42282370	device

$ incus restart adb
$ incus exec adb -- adb devices
* daemon not running; starting now at tcp:5037
* daemon started successfully
List of devices attached
42282370	device
$

Obviously, when you want to perform more work with adb, you can just get a shell into the container.

$ incus exec adb -- sudo --login --user debian
debian@adb:~$ adb devices
List of devices attached
42285120	device

debian@adb:~$ adb shell
komodo:/ $ exit
debian@adb:~$ logout
$

Conclusion

If you setup your system so that only a designated container can have access to your Android phone with USB debugging, then you get a bit better setup in terms of security.

How to share a folder between a host and a container in Incus

Simos Xenitellis — Thu, 22 Aug 2024 13:08:33 +0000

Incus is a manager for virtual machines and system containers.

In this post we are going to see how to conveniently share a folder between the host and an Incus container. The common use-case is that you want to share directly files between the host and a container and you want the file ownership to be handled well. Note that in a container the UID and GID do not correspond to the same values as on the host.

Therefore, we are looking on how to share storage between the host and one or more Incus containers. The other case that we look into earlier, is to share storage between containers that has been allocated on the Incus storage pool.

Quick answer

incus config device add mycontainer mysharedfolder disk source=/home/myusername/SHARED path=/SHARED shift=true

Quick answer
Background
Somewhat sharing folders between the host and Incus containers
How to properly share a folder between a host and a container in Incus
Incus config device commads
Conclusion

Background

On a Linux system the User ID (UID) and Group ID (GID) values are generally in the range of 0 (for root) to 65534 (for nobody). You can verify this by having a look at your /etc/passwd file on your system. In this file, each line is a different account; either a system account or a user account. Such a sample line is the following. There are several fields, separated by a colon character (:). The first field is the user name (here, root). The third field is the numeric user ID (UID) with the value of 0. The fourth field is the numeric group ID (GID) with the value of 0 as well. This is the root account.

root:x:0:0:root:/root:/bin/bash

Let’s do another one. The default user account in Debian and Debian-derived Linux distributions. The username in this Linux installation is myusername, the UID is 1000 and the GID is 1000 as well. This value of 1000 is quite common between Linux distributions.

myusername:x:1000:1000:User,,,:/home/user:/bin/bash

And now this one is the last one we will do. This is a special account with username nobody, and UID/GID at 65534. The purpose of this account is to be used for resources that somehow do not have a valid ID, or for processes and services that are expected to have the least privileges. In an Incus container you will see nobody and nogroup if you shared a folder and the translation of the IDs between the host and the container did not work well or did not happen at all.

nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin

To share a directory from the host to your Incus container(s), you add a disk device using incus config device. Here, mycontainer is the name of the container. And mysharedfolder is the name of the share which is only visible in Incus. You specify the source (an existing folder on the host) and the path (a folder in the container). However, there’s one big issue here. There’s no translation between the UID and GID of the files and directories in that folder.

incus config device add mycontainer mysharedfolder disk source=/home/myusername/SHARED path=/SHARED

Here’s on the host and in the container and then again on the host. In the last command it shows an empty folder. I think it’s possible to view the contents of SHARED in the container from the viewpoint of the host. It would require a smart use of the nsenter command to enter the proper namespace which I am not sure yet how to do for ZFS storage pools. If it worked, it would show that UID of myusername in the container from the viewpoint of the host, it would be something like 1001000 (Base ID 100000 plus 1000 for the first non-root account). That is, if the container has files with UID/GID outside of its range of 100000 – 165535, those files are not accessible from within the container.

$ ls -l SHARED/
total 0
-rw-rw-r-- 1 myusername myusername 0 Ιουλ 30 19:26 one
-rw-rw-r-- 1 myusername myusername 0 Ιουλ 30 19:26 two
-rw-rw-r-- 1 myusername myusername 0 Ιουλ 30 19:26 three
$ incus exec mycontainer -- ls -l /SHARED/
total 0
-rw-rw-r-- 1 nobody nogroup 0 Jul 30 16:26 one
-rw-rw-r-- 1 nobody nogroup 0 Jul 30 16:26 three
-rw-rw-r-- 1 nobody nogroup 0 Jul 30 16:26 two
$ sudo ls -l /var/lib/incus/containers/mycontainer/rootfs/SHARED
total 0
$

This was a good exercise to show the common mistake in sharing a folder from the host to the container. You can now remove the disk device and do it again properly.

$ incus config device remove mycontainer mysharedfolder
Device mysharedfolder removed from mycontainer
$

The shared folder requires some sort of automated UID/GID shifting so that from the point of view of the container, it has valid (within range) values for the UID/GID. This is achieved with the parameter shift=true when we create the Incus disk device.

Let’s see a full example. We create a container and then create a folder on the host, which we call SHARED. Then, in that folder we create three empty files using the touch command. We are being fancy here. Then, we create the Incus disk device to share the folder into the container, and enable shift=true.

$ incus launch images:ubuntu/24.04/cloud mycontainer
Launching mycontainer
$ mkdir SHARED
$ touch SHARED/one SHARED/two SHARED/there
$ incus config device add mycontainer mysharedfolder disk source=/home/myusername/SHARED path=/SHARED shift=true
Device mysharedfolder added to mycontainer
$

where

incus config device, the Incus command to configure devices.
add, we add a device.
mycontainer, the name of the container.
mysharedfolder, the name of the shared folder. This is only visible from the host, and it’s just any name. We need a name so that we can specify the device when we want to perform further management.
disk, this is a disk device. Currently, Incus supports 12 types of devices.
source=/home/myusername/SHARED, the absolute path to the source folder. We type source= and then source folder, no spaces in between. The source folder (which is on the host) must already exist.
path=/SHARED, the absolute path to the folder in the container.
shift=true, the setting that automagically performs the necessary UID/GID translation.

In some cases, for example when your Linux kernel on the Incus host is old, the shift=true setting may not work. Or, some filesystems may not support it. I leave it to you to report back any cases where this did not work. In the comments below.

Let’s verify that everything work OK. First we see the files on the host. In my case, both the user myusername and group myusername have UID and GID 1000. In the container (which was images:ubuntu/24.04/cloud) they also have the same UID/GID of 1000, but for this Ubuntu runtime the default username with UID 1000 is ubuntu, and the default group with GID 1000 is lxd. These are just names and are not important. If you are still unsure, then use ls with the added parameter --numeric-uid-gid to show the numeric IDs. In both cases below, the UID and GID are 1000.

$ ls -l SHARED/
total 0
-rw-rw-r-- 1 myusername myusername 0 Ιουλ 30 19:26 one
-rw-rw-r-- 1 myusername myusername 0 Ιουλ 30 19:26 two
-rw-rw-r-- 1 myusername myusername 0 Ιουλ 30 19:26 three
$ incus exec mycontainer -- ls -l /SHARED/
total 0
-rw-rw-r-- 1 ubuntu lxd 0 Jul 30 16:26 one
-rw-rw-r-- 1 ubuntu lxd 0 Jul 30 16:26 two
-rw-rw-r-- 1 ubuntu lxd 0 Jul 30 16:26 three
$

If we had created an images:debian/12/cloud container, here is how the files would look like. The username with UID 1000 is debian, and the group with GID 1000 is netdev.

$ incus exec mycontainer -- ls -l /SHARED/
total 0
-rw-rw-r-- 1 debian netdev 0 Jul 30 16:26 one
-rw-rw-r-- 1 debian netdev 0 Jul 30 16:26 three
-rw-rw-r-- 1 debian netdev 0 Jul 30 16:26 two
$

You can create files and subfolders in the shared folder, either on the host or in the container. You can also share it between more containers.

Incus config device commads

Let’s have a look at the incus config device commands.

$ incus config device 
Usage:
  incus config device [flags]
  incus config device [command]

Available Commands:
  add         Add instance devices
  get         Get values for device configuration keys
  list        List instance devices
  override    Copy profile inherited devices and override configuration keys
  remove      Remove instance devices
  set         Set device configuration keys
  show        Show full device configuration
  unset       Unset device configuration keys

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

Use "incus config device [command] --help" for more information about a command.
$

We are going to use some of those commands. First, we list the disk devices. There’s currently only one disk device, mysharedfolder. We then show the disk device. We get the list of parameters, which we can get individually and even set them to different values. We then get the value of the path parameter. Finally, we remove the disk device. We have not shown how to override, set and unset.

$ incus config device list mycontainer
mysharedfolder
$ incus config device show mycontainer
mysharedfolder:
  path: /SHARED
  shift: "true"
  source: /home/myusername/SHARED
  type: disk
$ incus config device get mycontainer mysharedfolder path
/SHARED
$ incus config device remove mycontainer mysharedfolder 
Device mysharedfolder removed from mycontainer
$

Conclusion

We have seen how to create disk devices on Incus containers in order to share a folder from the host to one or more containers. Using shift=true we can take care of the translation of the UIDs and GIDs. I am interested in corner cases where these do not work. It would help me fix this post, and eventually move it to the official documentation of Incus.

Running OCI images (i.e. Docker) directly in Incus

Simos Xenitellis — Fri, 12 Jul 2024 16:50:21 +0000

One of the cool new features in Incus 6.3 is the ability to run OCI images (such as those for Docker) directly in Incus.

You can certainly install the Docker packages in an Incus instance but that would put you in a situation of running a container in a container. Why not let Docker be a first-class citizen in the Incus ecosystem?

Note that this feature is new to Incus, which means that if you encounter issues, please discuss and report them.

Launching the docker.io nginx OCI container image in Incus.

Background
Prerequisites
Adding the Docker repository to Incus
Launching a Docker image in Incus
Discussion
Troubleshooting

Background

In Incus you typically run system containers, which are containers that have been setup to resemble a virtual machine (VM). That is, you launch a system container in Incus, and this system container keeps running until you stop it. Just like with VMs.

In contrast, with Docker you are running application containers. You launch the Docker container with some configuration to perform a task, the task is performed, and the container stops. The task might also be something long-lived, like a Web server. In that case, the application container will have a longer lifetime. With application containers you are thinking primarily about tasks. You stop the task and the container is gone.

Prerequisites

You need to install Incus 6.3. If you are using Debian or Ubuntu, you would select the stable repository of Incus.

$ incus version
Client version: 6.3
Server version: 6.3
$

Adding the Docker repository to Incus

The container images from Docker follow the Open Container Image (OCI) format. There is also a special way to access those images through the Docker Hub Container Image Repository, which is distinctive from the other ways supported by Incus.

We will be adding (once only) a remote for the Docker repository. A remote is a configuration to access images from a particular repository of such images. Let’s see what we already have. We run incus remote list which invokes the list command for the functionality about remotes (incus remote). There are two remotes, the images which is the standard repository for container images and virtual machine images for Incus. And then there is local, which is the remote of the local installation of Incus. Every installation of Incus has such a default remote.

$ incus remote list
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
|      NAME       |                URL                 |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC | GLOBAL |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
| images          | https://images.linuxcontainers.org | simplestreams | none        | YES    | NO     | NO     |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix://                            | incus         | file access | NO     | YES    | NO     |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
$

The Docker repository has the URL https://docker.io and it is accessed through a different protocol. It is called oci.

Therefore, to add the Docker repository, we need to run incus remote add with the appropriate parameters. The URL is https://docker.io and the --protocol is oci.

$ incus remote add docker https://docker.io --protocol=oci
$

Let’s list again the available Incus remotes. The docker remote has been added successfully.

$ incus remote list
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
|      NAME       |                URL                 |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC | GLOBAL |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
| docker          | https://docker.io                  | oci           | none        | YES    | NO     | NO     |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
| images          | https://images.linuxcontainers.org | simplestreams | none        | YES    | NO     | NO     |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix://                            | incus         | file access | NO     | YES    | NO     |
+-----------------+------------------------------------+---------------+-------------+--------+--------+--------+
$

If we ever want to remove a remote called myremote, we would incus remote remove myremote.

Launching a Docker image in Incus

When you launch (install and run) a container in Incus, you use incus launch with the appropriate parameters. In this first example, we launched the image hello-world, which is one of the Docker official images. As an image, it runs, printing some text and then it stops. In this case we used the parameter --console in order to see the text output. Finally, we use the --ephemeral parameter that would automatically delete the container image as soon as it stops. Ephemeral (εφήμερο) is a Greek word, meaning that it lasts only a brief time. Both these two additional parameters are not essential but are helpful in this specific case.

$ incus launch docker:hello-world --console --ephemeral
Launching the instance
Instance name is: best-feature                       

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

$

Note that we did not specify a name for the container. Astonishingly, Incus randomly selected the name best-feature with some editorial help. Since this is an ephemeral container and the specific image hello-world is short-lived, it is gone in a flash. Indeed, if you then run incus list, the Docker container is not found because it has been auto-deleted.

Let’s try another Docker image, the official nginx Docker image. It launches the nginx image which serves an empty nginx Web server. We need to run incus list and search for the IP address that was given to the container. Then, we can view the default page of the Web server in our Web browser.

$ incus launch docker:nginx --console --ephemeral
Launching the instance
Instance name is: best-feature                               
To detach from the console, press: +a q
2024/07/12 16:31:59 [notice] 21#21: using the "epoll" event method
2024/07/12 16:31:59 [notice] 21#21: nginx/1.27.0
2024/07/12 16:31:59 [notice] 21#21: built by gcc 12.2.0 (Debian 12.2.0-14) 
2024/07/12 16:31:59 [notice] 21#21: OS: Linux 6.5.0-41-generic
2024/07/12 16:31:59 [notice] 21#21: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/07/12 16:31:59 [notice] 21#21: start worker processes
2024/07/12 16:31:59 [notice] 21#21: start worker process 45
2024/07/12 16:31:59 [notice] 21#21: start worker process 46
2024/07/12 16:31:59 [notice] 21#21: start worker process 47
2024/07/12 16:31:59 [notice] 21#21: start worker process 48
2024/07/12 16:31:59 [notice] 21#21: start worker process 49
2024/07/12 16:31:59 [notice] 21#21: start worker process 50
2024/07/12 16:31:59 [notice] 21#21: start worker process 51
2024/07/12 16:31:59 [notice] 21#21: start worker process 52
2024/07/12 16:31:59 [notice] 21#21: start worker process 53
2024/07/12 16:31:59 [notice] 21#21: start worker process 54
2024/07/12 16:31:59 [notice] 21#21: start worker process 55
2024/07/12 16:31:59 [notice] 21#21: start worker process 56
10.10.10.1 - - [12/Jul/2024:16:33:29 +0000] "GET / HTTP/1.1" 200 615 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "-"
...
^C
...
2024/07/12 16:39:10 [notice] 21#21: signal 29 (SIGIO) received
2024/07/12 16:39:10 [notice] 21#21: signal 17 (SIGCHLD) received from 55
2024/07/12 16:39:10 [notice] 21#21: worker process 55 exited with code 0
2024/07/12 16:39:10 [notice] 21#21: exit
Error: stat /proc/-1: no such file or directory
$

Discussion

How long is the lifecycle time of a minimal Docker container?

How long does it take to launch the docker:hello-world container? I prepend time to the command, and check the stats at the end of the execution. It takes about four seconds to launch and run a simple Docker container. On my system ~~and my Internet connection~~ (it’s cached).

$ time incus launch docker:hello-world --console --ephemeral
...
real	0m3,956s
user	0m0,016s
sys	0m0,016s
$

How long does it take to repeatedly run a minimal Docker container?

We are removing the --ephemeral option. We launch the container and we give some relevant name, mydocker. This container remains after execution. We can see that after launching it, the container stays in a STOPPED state. We then incus start the container with the command time incus start mydocker --console, and we can see that it takes a bit more than half a second to complete the execution.

$ incus launch docker:hello-world mydocker --console
Launching mydocker

Hello from Docker!
...
$ incus list mydocker
+----------+---------+------+------+-----------------+-----------+
|   NAME   |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+----------+---------+------+------+-----------------+-----------+
| mydocker | STOPPED |      |      | CONTAINER (APP) | 0         |
+----------+---------+------+------+-----------------+-----------+
$ time incus start mydocker --console

Hello from Docker!
...

Hello from Docker!
...
real	0m0,638s
user	0m0,003s
sys	0m0,013s
$

Are the container images cached?

Yes, they are cached. When you launch a container for the first time, you can visibly the downloading of the image components. Also, if you run incus image list, you can see the cached Docker.io images in the output.

Troubleshooting

Error: Can’t list images from OCI registry

You tried to list the images of the Docker Hub repository. Currently this is not supported. Technically it could be supported though the repository has more than ten thousand images. Using incus list without parameters may not make sense as the output would require to download the full list of images from the repository. Searching for images does make sense, but if you can search, you should be able to list as well. I am not sure what’s the maintainer’s view on this.

$ incus image list docker:
Error: Can't list images from OCI registry
$

As a workaround, you can simply locate the image name from the Website at https://hub.docker.com/search?q=&image_filter=official

Error: stat /proc/-1: no such file or directory

I ran many many instances of Docker containers and in a few cases I got the above error. I do not know what it is and I am adding it here in case someone manages to replicate. It feels that it’s some kind of race condition.

Failed getting remote image info: Image not found

You have configured Incus properly and you definitely have Incus version 6.3 (both the client and the server). But still, Incus cannot find any Docker image, not even hello-world.

This can happen if you are using a packaging of Incus that does not include the skopeo and umoci packages. If you are using the Zabbly distribution of Incus, these programs are included in the Incus packaging. Therefore, if you are using alternative packaging for Incus, you can manually install the versions of those packages as provided by your Linux distribution.

How to install and setup the Incus Web UI

Simos Xenitellis — Thu, 28 Mar 2024 22:16:39 +0000

Incus is a manager for virtual machines (VM) and system containers. There is also an Incus support forum.

Typically you would use the incus command-line interface (CLI) client to get access to the Incus manager and perform the tasks for the full life-cycle of the virtual machines and system containers.

In this post we see how to install and setup the Incus Web UI. Just like the incus CLI tool that gets access to the REST API of the Incus manager (through a Unix socket or HTTPS), the Incus Web UI does the same over HTTPS. I assume that you have already installed and setup Incus.

Prerequisites
Installing the Incus Web UI package
Preparing Incus to serve the Web UI
Getting the browser to authenticate to the server
Using the Incus UI
Conclusion
Tips and Tricks
Troubleshooting

Prerequisites

You should already have a installation of Incus. If you do not have yet, see the official documentation on Incus installation and Incus migration, or my prior posts on Incus installation and Incus migration.

Installing the Incus Web UI package

The Incus Web UI package is incus-ui-canonical. We install it. By installing the package, we can enable Incus to serve the necessary Web pages (from /opt/incus/ui) so that we can connect with our browser and manage Incus itself. Note that currently the incus-ui-canonical package is provided by the Zabbly repository.

sudo apt install -y incus-ui-canonical

Preparing Incus to serve the Web UI

By default Incus is not listening to a Web port so that we can access directly through the browser. We need to enable first Incus to activate access to the Web browser. By default there is no configuration with incus config show.

debian@myincus:~$ incus config show 
config: {}
debian@myincus:~$

We activate the Incus Web server, selecting the port number 8443. You are free to select another one, if you need to. We set core.https_address to :8443. This information appears in the incus config output.

debian@myincus:~$ incus config set core.https_address :8443
debian@myincus:~$ incus config show 
config:
  core.https_address: :8443
debian@myincus:~$

Let’s verify that Incus is now listening to port 8443. Yes, it does. On all interfaces (because of the *).

debian@myincus:~$ sudo apt install -y lsof
...
debian@myincus:~$ sudo lsof -i :8443
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
incusd  8338 root    8u  IPv6  29751      0t0  TCP *:8443 (LISTEN)
debian@myincus:~$

This is HTTPS, where are the certificate and the server key (private key)?

debian@myincus:~$ sudo ls -l /var/lib/incus/server.key /var/lib/incus/server.crt
-rw-r--r-- 1 root root 753 Mar 28 18:54 /var/lib/incus/server.crt
-rw------- 1 root root 288 Mar 28 18:54 /var/lib/incus/server.key
debian@myincus:~$ sudo openssl x509 -in /var/lib/incus/server.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            22:05:f1:14:f2:82:43:68:44:5e:1c:42:4c:28:5b:5c
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: O = Linux Containers, CN = root@myincus
        Validity
            Not Before: Mar 28 18:54:17 2024 GMT
            Not After : Mar 26 18:54:17 2034 GMT
        Subject: O = Linux Containers, CN = root@myincus
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:fb:cd:b6:b2:25:55:68:a5:33:75:48:4c:b0:7a:
                    2f:e9:c0:16:af:6f:b2:36:f9:19:6e:b0:86:bf:d1:
                    9f:07:16:b1:26:8b:75:36:f2:fc:02:38:c7:fa:25:
                    39:01:6c:bb:48:a9:4f:57:0d:af:e1:0f:a3:cf:b1:
                    7c:a2:d9:46:77:e7:94:c7:00:1a:d0:5f:5f:93:d8:
                    11:39:8d:16:0e:d0:62:98:81:93:da:ec:b8:70:24:
                    f2:c4:da:91:0f:f8:8e
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:myincus, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1
    Signature Algorithm: ecdsa-with-SHA384
    Signature Value:
        30:64:02:30:15:f4:fa:7b:d6:52:79:d4:c9:27:b9:d6:6c:90:
        f7:0e:13:83:15:ac:af:cd:c5:f2:48:08:99:7f:7b:94:55:06:
        81:95:80:5f:0a:21:17:82:61:ac:5a:b6:5f:b8:49:b3:02:30:
        62:a3:92:66:da:ce:7c:01:49:7e:38:16:c6:16:b3:cb:aa:3d:
        1d:3f:63:12:93:e8:a1:0b:55:f0:80:99:d5:80:8a:a3:a6:2e:
        3d:68:90:a6:dc:55:29:0b:36:80:36:72

debian@myincus:~$

Note that this is a self-signed certificate. Chrome, Firefox and other browsers will complain; you can still accept to continue but it will show a broken padlock at the address bar. If you wish, you can replace these with proper certificates so that the padlock is intact. To do so, once you replace the server key and the server certificate with actual values, restart Incus. If, however, you are running an Incus cluster, you must use incus cluster update-certificate instead to update them. Note that a common alternative to dealing with Incus certificates, is to use a reverse-proxy; you get the reverse-proxy to use a proper certificate and leave Incus as is.

At this point Incus is configured. We can continue with the next step where we get the client (our browser) to be authenticated to the server.

Getting the browser to authenticate to the server

Visit the URL of your Incus server with your browser. At first you will likely confronted with a message that the server certificate is not accepted (Warning: Potential Security Risk Ahead). Click to Accept and continue. Then, you are presented with the following screen that asks you to login. You are authenticated to the Incus server through user certificates. You are prompted here to do just that. Your browser will create

a user certificate to be installed into Incus (incus-ui.crt)
the same user certificate with a private key that will be setup in your browser(s) (incus-ui.pfx).

Click on Create a new certificate.

Creating a new certificate.

Now click on Generate to get your browser to generate the private key and the certificate.

You are asked whether you want to protect the certificate with a password. In our case we click on Skip because we do not want to encrypt the private key with a password. By clicking on Skip, the private key is still generated but it is not getting encrypted.

At this point the browser generated incus-ui.crt, which is the user certificate to install in Incus. In the following we added the user certificate to Incus.

debian@myincus:~$ incus config trust list
+------+------+-------------+-------------+-------------+
| NAME | TYPE | DESCRIPTION | FINGERPRINT | EXPIRY DATE |
+------+------+-------------+-------------+-------------+
debian@myincus:~$ incus config trust add-certificate incus-ui.crt
debian@myincus:~$ incus config trust list
+--------------+--------+-------------+--------------+----------------------+
|     NAME     |  TYPE  | DESCRIPTION | FINGERPRINT  |     EXPIRY DATE      |
+--------------+--------+-------------+--------------+----------------------+
| incus-ui.crt | client |             | b89b80eb4c89 | 2026/12/23 21:08 UTC |
+--------------+--------+-------------+--------------+----------------------+
debian@myincus:~$

The two files have been generated. We are adding incus-ui.crt to Incus, and incus-ui.pfx to the Web browser.

The page above has instructions on how to add the user certificate to Firefox, Chrome, Edge and macOS. For example, for the case of Firefox, type the following to the address bar and press Enter. Alternatively, go to Settings→Privacy & Security→Certificates. There, click on View Certificates… and select the Your Certificates tab. Finally, click to Import… the incus-ui.pfx certificate file.

about:preferences#privacy

This is found in Firefox under Settings→Privacy & Security→Certificates.

When you add the incus-ui.pfx user certificate in Firefox, it will appear as in the following screenshot.

The incus-ui.pfx certificate has been added to this instance of Firefox.

Subsequently, switch back to the Firefox tab with the Incus UI page and you are shown the following prompt to get your browser to send the user certificate to the Incus manager in order to get authenticated, and be able to manage Incus through the Web. Click on OK.

You are prompted to identify yourself to Incus UI in order to be able to manage the Incus installation.

Finally, you are able to manage Incus over the Web with Incus UI. The Web page loads up and you can perform all tasks that you can do with the incus command-line client.

Your browser is now authenticated through your user certificate and you can manage Incus over the Web with Incus UI.

Using the Incus UI

We click on Create Instance to create a first instance. We select from the list which image to use, then click to Create and start.

Creating an instance and starting it.

While the instance is created, you are updated with the different steps that take place. In the end, the instance is successfully launched.

The instance has been created and is running.

Conclusion

With Incus UI you are able to go through all the workflow of managing Incus instances through your Web browser. Incus UI has been implemented as a stateless Web application, which means that no information are stored on the browser. For example, the browser does not maintain a database with the created instances; the state is maintained on Incus.

In this post we saw how to setup Incus UI with SSL/TLS authentication. It’s also possible to setup Incus UI to use Single Sign-On (SSO). Here is a tutorial on how to setup Incus UI with Open-ID Connect (OIDC).

There are a few more UI Web applications for Incus, including lxops. At some point in the future I expect to cover them as well.

Tips and Tricks

How to make the Incus port accessible to localhost only

The address has the format of <ip address>:<port>. You can specify localhost (127.0.0.1) for the part of the IP address. By doing so, Incus will only bind to localhost and listen to local connections only.

debian@myincus:~$ incus config show
config:
  core.https_address: :8443
debian@myincus:~$ incus config set core.https_address 127.0.0.1:8443
debian@myincus:~$ incus config show
config:
  core.https_address: 127.0.0.1:8443
debian@myincus:~$ sudo lsof -i :8443
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
incusd  8338 root    8u  IPv4  30315      0t0  TCP localhost:8443 (LISTEN)
debian@myincus:~$

What’s in `incus-ui.crt` and `incus-ui.pfx`?

You can use openssl to decode both files. This is an RSA 2048-bit certificate using the SHA-1 hash function.

$ openssl x509 -in incus-ui.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            01:12:00:11:07:65:00:03:00:10:00:41:00:04:09:11
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C = AU, ST = Some-State, O = Incus UI 10.10.10.98 (Browser Generated)
        Validity
            Not Before: Mar 28 21:08:58 2024 GMT
            Not After : Dec 23 21:08:58 2026 GMT
        Subject: C = AU, ST = Some-State, O = Incus UI 10.10.10.98 (Browser Generated)
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ce:f8:1d:67:e1:a3:f5:1a:16:b6:26:63:8f:32:
                    42:99:0d:af:86:8b:18:49:1a:4b:8e:ab:68:e1:04:
                    ba:24:dd:e6:27:d5:df:7a:13:cf:16:b3:33:28:89:
                    e0:ab:c8:dc:c1:2a:0a:de:ed:26:3a:77:74:dd:42:
                    1c:e2:22:fc:a5:a5:68:c1:c9:3b:4d:12:15:27:ae:
                    c6:50:ec:dc:f1:0a:ba:00:0c:83:d0:0d:0f:81:90:
                    4e:30:43:cb:45:bf:e2:e9:17:39:40:3b:95:8b:8b:
                    18:e9:59:51:fc:9a:7a:80:e4:73:b3:54:bd:ff:1c:
                    7c:81:75:16:e3:6f:3a:56:9b:0f:a3:73:55:45:03:
                    d8:fb:f3:34:4c:60:4f:f2:67:9f:66:ea:29:29:78:
                    6c:66:05:d6:7d:96:cd:0f:2b:4b:9c:71:2c:09:6f:
                    e2:b4:23:d0:5d:d0:fe:b0:6a:b1:58:5e:d7:b5:47:
                    9e:aa:47:34:f8:7d:e1:ed:fe:bf:97:3d:99:49:42:
                    af:e2:e5:b3:c5:1e:58:b1:98:01:db:8f:25:9f:f8:
                    d9:03:02:06:f9:99:0a:3a:a1:70:9d:fe:64:0d:c2:
                    d8:cc:f0:1c:53:e4:31:4c:78:12:c2:fd:72:23:6a:
                    f4:7e:41:f9:d5:df:6b:ad:2c:52:29:d0:7f:eb:65:
                    64:0f
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha1WithRSAEncryption
    Signature Value:
        28:b3:5c:48:64:8c:23:82:dd:e2:05:6a:9d:18:dd:43:f4:07:
        e6:be:1e:80:b7:f9:0c:0f:3d:cd:b8:bd:7b:55:7e:36:6d:74:
        24:d5:69:b2:24:51:3a:2d:c5:95:68:b5:dc:27:d5:83:d9:bc:
        cb:d0:fd:55:24:63:7d:c6:65:9b:f1:b3:9d:f7:b4:4e:ba:83:
        eb:bf:f5:d0:f6:95:2d:7b:90:4e:d3:89:ac:f0:87:e6:fa:9d:
        f6:ea:c2:42:f2:15:17:74:5c:e4:3c:ed:1a:42:3c:e7:04:aa:
        65:42:3e:75:5c:24:8e:52:85:0d:4b:b2:e2:ec:fa:57:4a:68:
        35:4b:8f:3c:13:fc:15:09:80:5a:b1:c8:e0:22:f5:69:25:4b:
        46:8b:e0:b9:e1:3a:f5:0c:40:d2:c3:75:9c:79:9a:aa:68:9b:
        21:36:ed:67:cb:6d:fc:bc:f0:0b:5a:2b:1a:4c:73:67:c5:79:
        b6:27:b9:58:d0:c7:ea:84:21:bf:f4:7c:44:11:d7:88:ab:1d:
        e4:53:c9:10:cd:e6:b8:5a:7a:92:73:a8:1e:fe:1c:2e:dc:e8:
        7e:3d:e9:a2:6d:26:5a:09:40:a1:3e:51:40:8b:da:57:37:9a:
        8d:0e:d8:cf:c1:0a:b1:0b:95:53:05:41:29:39:af:93:9b:aa:
        10:af:a1:6c
$

For the incus-ui.pfx file, we first convert to the PEM format, then print the contents. The PFX file contains the certificate (the same that was added earlier to Incus) along with the private key.

$ openssl pkcs12 -in incus-ui.pfx -out incus-ui.pem -noenc
Enter Import Password:
$ cat incus-ui.pem 
Bag Attributes
    localKeyID: 3A 23 25 F7 56 4D 71 B8 FB FD 72 90 2D A1 F3 B8 2F 01 5E 92 
    friendlyName: Incus-UI
subject=C = AU, ST = Some-State, O = Incus UI 10.10.10.98 (Browser Generated)
issuer=C = AU, ST = Some-State, O = Incus UI 10.10.10.98 (Browser Generated)
-----BEGIN CERTIFICATE-----
MIIDMjCCAhqgAwIBAgIQARIAEQdlAAMAEABBAAQJETANBgkqhkiG9w0BAQUFADBV
MQswCQYDVQQGEwJBVTETMBEGA1UECBMKU29tZS1TdGF0ZTExMC8GA1UEChMoSW5j
dXMgVUkgMTAuMTAuMTAuOTggKEJyb3dzZXIgR2VuZXJhdGVkKTAeFw0yNDAzMjgy
MTA4NThaFw0yNjEyMjMyMTA4NThaMFUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIEwpT
b21lLVN0YXRlMTEwLwYDVQQKEyhJbmN1cyBVSSAxMC4xMC4xMC45OCAoQnJvd3Nl
ciBHZW5lcmF0ZWQpMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzvgd
Z+Gj9RoWtiZjjzJCmQ2vhosYSRpLjqto4QS6JN3mJ9XfehPPFrMzKIngq8jcwSoK
3u0mOnd03UIc4iL8paVowck7TRIVJ67GUOzc8Qq6AAyD0A0PgZBOMEPLRb/i6Rc5
QDuVi4sY6VlR/Jp6gORzs1S9/xx8gXUW4286VpsPo3NVRQPY+/M0TGBP8mefZuop
KXhsZgXWfZbNDytLnHEsCW/itCPQXdD+sGqxWF7XtUeeqkc0+H3h7f6/lz2ZSUKv
4uWzxR5YsZgB248ln/jZAwIG+ZkKOqFwnf5kDcLYzPAcU+QxTHgSwv1yI2r0fkH5
1d9rrSxSKdB/62VkDwIDAQABMA0GCSqGSIb3DQEBBQUAA4IBAQAos1xIZIwjgt3i
BWqdGN1D9Afmvh6At/kMDz3NuL17VX42bXQk1WmyJFE6LcWVaLXcJ9WD2bzL0P1V
JGN9xmWb8bOd97ROuoPrv/XQ9pUte5BO04ms8Ifm+p326sJC8hUXdFzkPO0aQjzn
BKplQj51XCSOUoUNS7Li7PpXSmg1S488E/wVCYBascjgIvVpJUtGi+C54Tr1DEDS
w3WceZqqaJshNu1ny238vPALWisaTHNnxXm2J7lY0MfqhCG/9HxEEdeIqx3kU8kQ
zea4WnqSc6ge/hwu3Oh+PemibSZaCUChPlFAi9pXN5qNDtjPwQqxC5VTBUEpOa+T
m6oQr6Fs
-----END CERTIFICATE-----
Bag Attributes
    localKeyID: 3A 23 25 F7 56 4D 71 B8 FB FD 72 90 2D A1 F3 B8 2F 01 5E 92 
    friendlyName: Incus-UI
Key Attributes: 
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDO+B1n4aP1Gha2
JmOPMkKZDa+GixhJGkuOq2jhBLok3eYn1d96E88WszMoieCryNzBKgre7SY6d3Td
QhziIvylpWjByTtNEhUnrsZQ7NzxCroADIPQDQ+BkE4wQ8tFv+LpFzlAO5WLixjp
WVH8mnqA5HOzVL3/HHyBdRbjbzpWmw+jc1VFA9j78zRMYE/yZ59m6ikpeGxmBdZ9
ls0PK0uccSwJb+K0I9Bd0P6warFYXte1R56qRzT4feHt/r+XPZlJQq/i5bPFHlix
mAHbjyWf+NkDAgb5mQo6oXCd/mQNwtjM8BxT5DFMeBLC/XIjavR+QfnV32utLFIp
0H/rZWQPAgMBAAECggEBAMm1N/tpBgC291F4YmlJg2xk0R8f6oA8V0zpMyKyF7Qc
atWB8/Wm3pnx9bbZgRQKg1LiZYvTtgEfMM7+QuYFURMi/NB4DQpUyDdPd0mhPsbQ
WVH8mnqA5HOzVL3/HHyBdRbjbzpWmw+jc1VFA9j78zRMYE/yZ59m6ikpeGxmBdZ9
+uKyZ4U4/TORu2tadg9frtUl1HhkY1zGAxOyJUbCOVIbZF2iQt5zMZt4XLFhKgwh
jtDklc3dFIDigUZzpMgdLExLWi6CGT++cjJGpseM+QOAubSoCmT6eIs8qi9KpQhk
aZYBerWqBxswkmNGK4Zh+5gFvdW7EmEp128hATgYZGECgYEA7ckh3qL4Jg6FQA8+
UeEoaT2CvDI89HMJfFN2NvU1ZklqP9aDnPvMjui/h/8HtDeb+5FWFZHF1B9laJp3
HnGGt+98/aO9skdFQDiszclDNIHdpSqcD2LWkKz84QTWqTTkRAxJpgnW91oURtyh
WVH8mnqA5HOzVL3/HHyBdRbjbzpWmw+jc1VFA9j78zRMYE/yZ59m6ikpeGxmBdZ9
JSltWZtYemYzPTpZysocyRs5mD8CgYEA3tKviDreIR+TKT3FQoevyicXuwSn6ocH
2RTgJQF+Qyj+1ykQhwRQUD+axZGls5g2JgT+2gFIdUcAR9CN22rxLRbnIj645yGP
Ka4dVhNAZnz/olWgs4onoO0CnOGXAkVdyiBe9H/D1dkj5bqAfY1eov6khPMOyrDF
EXGi0e6uInbddI/sHUAAIIqJ4+knqwJIgxlzA9GFuzzt4oRLGMsoaClLYFCsrekJ
SF/w7DvhoDQo+JIrHuGX4hLgFLWOgp2WMWhbvgZ0P1PWcJukZ/jx7rJmkwKBgGa5
7x75NMtEiU3sInMnpw2ltDUOUnO3SRD1pNiqtZE05zg+wFXe0UAN8sa+/QutUtl4
WVH8mnqA5HOzVL3/HHyBdRbjbzpWmw+jc1VFA9j78zRMYE/yZ59m6ikpeGxmBdZ9
WB4dlVAsKZ7yMVRFG2dUNb7997TnLd9jXDcArSIS4q/uliXvvZFdc2TsQ/hSDolP
HzfNZ3XBo+EXeIFpmYW/rA13GQytLl5oDC28WaEhAoGBAL6acBqMflXUoWWVHZR7
0vNcJjtRTC13SGRoAKR/tT2kUqloz60bgWeVtggkFWTpPGgm6lmSuYvTnPeoHYDf
vLibVFGasTk8Y7Aji0V7rF4O
-----END PRIVATE KEY-----
$

How to configure Incus about the location of the Incus UI files?

When the Incus service starts running, it uses a hard-coded location for the web files of Incus UI. If that location is not the actual location of your files, then your browser will show the error 404 (files not found). In that case, you can setup your system so that the incusd service will be able to find the proper location by setting the INCUS_UI=/opt/incus/ui environment variable. You would need to look into your system’s configuration so that the Incus service will be able to be exposed to the environment variable. One location could be /etc/environment (not tested) or the service script that starts incusd.

Troubleshooting

Error: Unable to connect

You tried to access the IP address of the Incus server as (for example) https://192.168.1.10/ while you should have specified the IP address as well. The URL should look like https://192.168.1.10:8443/.

Error: Client sent an HTTP request to an HTTPS server

You tried to connect to the Incus server at an address (for example) http://192.168.1.10:8443/ but you omitted the s in https. Use https://192.168.1.10:8443/ instead.

Warning: Potential Security Risk Ahead

You are accessing the Incus server through the HTTPS address for the first time and the certificate has not been signed by a certification authority.

First attempt to access the Incus server over HTTPS with your browser.

Click on Advanced and select to Accept the risk and Continue. If you want to avoid this error message, you need to provide a server certificate that is accepted by your browser.

E: Unable to locate package incus-ui-canonical

You are trying to install incus-ui-canonical and the package is not found.

The package is available from the Zabbly repository (Debian and Ubuntu) and may not exist in the repositories of your Linux distribution.

See https://github.com/zabbly/incus?tab=readme-ov-file#installation on how to add the Zabbly repository.

Message {“type”:”sync”,”status”:”Success”,”status_code”:200,”operation”:””,”error_code”:0,”error”:””,”metadata”:[“/1.0”]}

You have setup both Incus and the Web UI package but when you visit the URL, the page does not load but instead you get output similar to the following. This can happen if there is a mismatch between the version of the Web UI and Incus.

In this specific case, the user had enabled the Ubuntu Expanded Security Maintenance (ESM) and the system enabled a separate version of Incus from the ESM package repository. Packages from that repository take precedence over other repositories.

To verify whether you are affected, run the following command that shows which version of Incus is installed on your system. If the three asterisks (***) are on the line that starts with 6.0.0-1ubuntu0.2+esm1, then the Incus package from the ESM repository is enabled and you are affected.

$ apt policy incus
incus:
  Installed: 1:6.12-ubuntu24.04-202505241744
  Candidate: 1:6.12-ubuntu24.04-202505241744
  Version table:
 *** 1:6.12-ubuntu24.04-202505241744 1001
        500 https://pkgs.zabbly.com/incus/stable noble/main amd64 Packages
        100 /var/lib/dpkg/status
     1:6.12-ubuntu24.04-202505220530 1001
        500 https://pkgs.zabbly.com/incus/stable noble/main amd64 Packages
     1:6.12-ubuntu24.04-202505151658 1001
        500 https://pkgs.zabbly.com/incus/stable noble/main amd64 Packages
     6.0.0-1ubuntu0.2+esm1 510
        510 https://esm.ubuntu.com/apps/ubuntu noble-apps-security/main amd64 Packages
     6.0.0-1ubuntu0.2 500
        500 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages
     6.0.0-1 500
        500 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages
$

A workaround is to reconfigure you Linux distribution so that your system prefers the latest version of Incus from the Zabbly repository.

$ sudo nano /etc/apt/preferences.d/incus-zabbly.pref
...
$ cat /etc/apt/preferences.d/incus-zabbly.pref
Package: incus
Pin: origin pkgs.zabbly.com
Pin-Priority: 1001
$ sudo apt update
$ sudo apt install --reinstall incus

Credits for this find to Yassine Zara at Incus WebUI returns an empty page.

Mi blog lah!

How to run Firecracker microVMs with vmsan on Incus

Table of Contents

Prerequisites

Setting up vmsan

Using vmsan

vmsan create to create a microVM

vmsan list to list the microVMs

vmsan connect to get a shell into a microVM

Running nginx in a microVM

Conclusion

How to run a Linux Desktop virtual machine on Incus

Table of Contents

Updates

Prerequisites

Cheat sheet

Availability of images

Booting a desktop Linux VM on Incus

Booting the Ubuntu desktop image on Incus

Booting the ArchLinux desktop image on Incus

Booting the OpenSUSE desktop image on Incus

Troubleshooting

I closed the desktop window but the VM is running. How do I get it back up?

Error: This console is already connected. Force is required to take it over.

Error: Instance is not running

I get no audio from the desktop VM! How do I get sound in the desktop VM?

How do I shutdown the desktop VM?

Error: Failed instance creation: The image used by this instance is incompatible with secureboot. Please set security.secureboot=false on the instance

Error: Failed instance creation: Failed creating instance record: Add instance info to the database: Failed to create “instances” entry: UNIQUE constraint failed: instances.project_id, instances.name

A networking guide for Incus

The setup

Default Incus networking

Technical details

Future considerations

How to recover or reconnect an Incus storage pool

Prerequisites

Setting up Incus, using a block storage volume

How to use your Android phone with ADB in an Incus container

Setting up the Android phone for USB Debugging

Setting up the host

Creating the container

Using adb in the Incus container

Stress testing adb in the Incus container

Conclusion

How to share a folder between a host and a container in Incus

Quick answer

Table of Contents

Background

Somewhat sharing folders between the host and Incus containers

How to properly share a folder between a host and a container in Incus

Incus config device commads

Conclusion

Running OCI images (i.e. Docker) directly in Incus

Table of Contents

Background

Prerequisites

Adding the Docker repository to Incus

Launching a Docker image in Incus

Discussion

How long is the lifecycle time of a minimal Docker container?

How long does it take to repeatedly run a minimal Docker container?

Are the container images cached?

Troubleshooting

Error: Can’t list images from OCI registry

Error: stat /proc/-1: no such file or directory

Failed getting remote image info: Image not found

How to install and setup the Incus Web UI

Table of Contents

Prerequisites

Installing the Incus Web UI package

Preparing Incus to serve the Web UI

Getting the browser to authenticate to the server

Using the Incus UI

Conclusion

Tips and Tricks

How to make the Incus port accessible to localhost only

What’s in incus-ui.crt and incus-ui.pfx?

How to configure Incus about the location of the Incus UI files?

Troubleshooting

Error: Unable to connect

Setting up `vmsan`

Using `vmsan`

`vmsan create` to create a microVM

`vmsan list` to list the microVMs

`vmsan connect` to get a shell into a microVM

What’s in `incus-ui.crt` and `incus-ui.pfx`?