How to Virtualize Your Router with OpenWrt and LXD

Is it time to replace your router? Are you unhappy with its performance or features?

If you answered yes to either of those questions, you should consider using a free, open-source router operating system. By virtualizing this appliance, you can improve the performance and extend the features of your current solution for almost no cost. Any computer with two network interfaces will do! The benefit of using a virtualized container for your router is that you can run it on a machine you already have, rather than buying another dedicated system for it.

OpenWrt is one of these operating systems that will allow you to easily configure a home router. It is conveniently provided as a container image in LXD's default repository. To genuinely replace something like a home router, the openwrt container and its host will need to run on the same network. The setup process to replace one network with another can get tricky, especially if you cannot allow for any down time. The following guide demonstrates how to setup a virtualized OpenWrt router with zero downtime, even if it is replacing your only network.

Why?

Better performance than a wifi-router appliance
More control over settings, software, and the base OS
Safer than trying to overwrite the firmware on your current router with OpenWrt

Requirements

Two physical network interfaces (minimum)
Tested on Ubuntu 22.04 with LXD snap, but will certainly work on most other platforms that provide LXD
16 MiB disk storage
128 MiB RAM

Important Caveat for LXD Clusters

For existing LXD clusters, it is recommended to use the same subnet range that your existing nodes use (their cluster.https_address) for the OpenWrt LAN subnet. If not, it will be necessary to rebuild the cluster if they are linked by the same subnet you are replacing.

If you are building a LXD cluster from scratch, be sure to specify the IP that the first node can be reached on within the desired LAN subnet before proceeding to setup OpenWrt.

If you are not clustering at all and only need to install LXD for the router, this setting is irrelevant. The below instructions will work for any of these use cases.

Host Setup

Let's assume the host has two NICs. One (eno1) is connected to the current router via ethernet, and the other (eno2) is unused.

First, create a virtual bridge on the host by adding this text to /etc/systemd/network/br0.netdev:
```
[NetDev]
Name=br0
SkipForwardingDelay=true
Kind=bridge
```
SkipForwardingDelay is important because it will allow the bridge interface to come up when the computer boots even if it cannot contact the gateway, which it won't be able to since the container wouldn't have started yet.

Then restart the networking service for the change to take effect:
```
sudo systemctl restart systemd-networkd
```
Add an interface in netplan and connect it to the bridge. The netplan configuration is stored in and .yaml file in /etc/netplan/.
```
network:
  ethernets:
    eno1:
...
    br0:
      ignore-carrier: true
      optional: true
      addresses: [ 10.10.0.10/16 ]
      nameservers:
        addresses: [ 10.10.0.1 ]
      routes:
      - to: default
        via: 10.10.0.1
...
```
Only add the part from br0 down, using the same number of spaces to indent each line. Change 10.10.0.10/16 to the IP and subnet range you want the host reached at, and 10.10.0.1 the IP of the router. The additional options of ignore-carrier and optional are critical for bringing up the interface when the host boots, even if it cannot connect, and skipping a several minute delay where it would try to make the connection.

Apply the change:

sudo netplan apply

Then, confirm the interface is listed as up with the expected IP:

ip a

...
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3e:40:e0:66:2c:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.10/16 brd 10.10.255.255 scope global br0
       valid_lft forever preferred_lft forever
...

Install LXD:
```
sudo snap install lxd
```
If you do not have the snap package manager, you can get it by installed snapd. On Debian-based distributions, this can be done with:
```
sudo apt install snapd
```
Launch the OpenWrt container:
```
lxc launch images:openwrt/22.03 openwrt
```
Passthrough the host bridge and the empty NIC (e.g. eno2) for now. This physical port will become the WAN port inside OpenWrt to which we will eventually connect our modem.
```
lxc config device add openwrt br0 nic nictype=bridged parent=br0 name=br0
lxc config device add openwrt eno2 nic nictype=physical parent=eno2 name=eno2
```

Container (OpenWrt) Setup

Open a shell prompt in the container and configure the network:

lxc exec openwrt -- busybox sh
vi /etc/config/network

Press i to type and paste these lines into the bottom of the file, changing the IPs accordingly:

config interface 'lan'
    option proto 'static'
    option ipaddr '10.10.0.1'
    option netmask '255.255.0.0'
    option device 'br-lan'

config device
    option name 'br-lan'
    option type 'bridge'
    list ports 'eno2'
    list ports 'br0'
    list ports 'eno1' # to be used for connecting additional machines later
    option bridge_empty '1'

Press Esc and type ZZ (with shift) to save and exit.

Restart the network and DHCP services:

service network restart
service dnsmasq restart

(Optional) This step is useful if you are running other LXD instances on the same LAN bridge that OpenWrt will be serving. Open the file /etc/rc.local with vi as above, and add this line above exit 0:
```
service dnsmasq restart
```
This will ensure that any other LXD containers connected to the OpenWrt LAN bridge will receive an IP immediately after a host reboot.
Restart the router:
```
reboot
```
You should now be able to ping the router IP (10.10.0.1) from the host and the host IP (10.10.0.10) from the OpenWrt container. Any additional LXD instances connected to the bridge should receive an IP automatically from DHCP. None of these instances can reach the internet yet from their br0 interface.
To allow outside connections to your new LAN, move the WAN cable (running from the modem to your old router) to the empty NIC (eno2). Check outbound connectivity from the router by pinging a public website and do the same from the host. If both can reach the outside world, everything should be in order.
If everything works, you can now add additional machines to your new LAN. In my case, I am reusing my old router as a wireless access point so that OpenWrt can handle wired and wireless connections. For this, I turned on the "access point" mode on my old router so that wireless clients would be managed completely by OpenWrt. If your router doesn't have this option, it is also fine to use as is.

You then want to take the cable from eno1, remove it from the router port it is currently in and place it in the old router's WAN port. The full setup will flow like this:
```
modem -> eno2 -> OpenWrt -> eno1 -> old router, switch, or access point
```
To allow the outbound port (eno1) to operate, pass it through to the OpenWrt container:
```
lxc config device add openwrt eno1 nic nictype=physical parent=eno1 name=eno1
```
And be sure to restart the networking services:
```
lxc exec openwrt -- service network restart
lxc exec openwrt -- service dnsmasq restart
```
If you cannot reach the outside world or local clients are not getting IP addresses on the new LAN subnet, see the next section.

Testing and Troubleshooting

Create another container on the same bridge:
```
lxc launch images:debian/11 test --network br0
```
See if it has an IP listed in lxc ls and trying to ping the router and/or the world from inside it:
```
lxc shell test
```
Run similar ping tests from the host and the OpenWrt container to see which piece is not communicating with the others.
For non-working clients, check the route table:
```
ip r
```
If the default route is set to something other than the new gateway, you will have to update it:
```
ip route add default via 10.10.0.1
ip route delete default via <old gateway>
```
This configuration will not persist through a reboot, but the client should pick it up from the DHCP server next time. If not, there is likely an issue with DHCP.
Monitor the logs in the OpenWrt web interface (connect via the gateway IP in a web browser) or from the shell:
```
logread
```
This should give you a good starting point for diagnosing more complex routing, DHCP, or DNS issues.