SmartOS to FreeBSD Bhyve migration

Situation: a failing and old SmartOS server at Hetzner running various lab serices as FreeBSD Bhyve guests in SmartOS zones. Not worth trying to fix it, time to commission a new server and migrate to it. FreeBSD's native Bhyve tooling is much better than when this journey began, so let's use that on the physical instead.

Additional consideration around networking: the SmartOS server has an IPv4 subnet routed to it and uses its IPv6 /64 for the guests as well as itself.

Hetzner do not permit randomly generated MACs on their network, so the physical server must route all this VM traffic to a backend bridge instead of bridging its physical interface. Getting this working on SmartOS was a little tricky.

Easy: Both SmartOS and FreeBSD run Bhyve VMs back by ZFS zvols. Both run the same architecture CPU. The virtual hardware exposed to the VMs is the same. Inside the VM the only change is their network configuration. On FreeBSD the port sysutils/vm-bhyve presents a simple interface to Bhyve VM configuration. We can script this easily enough, preserving VM hostnames, network config and transforming their zvol paths to match the target system.

Moderate: Regular ZFS migration dance: snapshot, send full dataset, snapshot, send incremental, repeat. Regular sysadmin stuff.

Harder: Work out how the backend network configuration without leaking MACs and generating Hetzner abuse mails. We landed at:

• bridge for the routed IPv4 subnet. private, no physical interface, bridge address is outbound gateway

• IPv6 /64 subnet on the same bridge, bridge address is outbound gateway

• net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=1

• for vm-bhyve define the bridge as manual

  switch_list="bridge0"
  type_bridge0="manual"
  bridge_bridge0="bridge0"
  

• allow all traffic to the routed IPv4 subnet and the IPv6 network, VMs do their own filtering

This means all packets routed out from the VMs via the bridge addresses traverse the FreeBSD server and appear on the physical interface using the correct MAC address.

Extra credit: by defining the bridge's IPv4 network wider than the actual routed subnet, and using a bridge address from this wider network, we can use all of the routed network addresses rather than losing three to the gateway, network and broadcast addresses.