VLAN for virtual machines
Inhaltsverzeichnis
Introduction
I wanted to update VLAN connections for virtual machines to newer technologies and put a question on unix.stackexchange. But I do not get any answer. It seems there is very little knowledge for this out there. So I decided to work on it by myself and document it here.
In gerneral I will look at three methods:
Preparation
I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: Setup KVM with console. I'm sitting on harley as host, my all day workstation. Now I start the virtual machine, login and show its interface setting:
harley$ virsh start --console deb9-test
login
deb9-test$ cat /etc/systemd/network/08-vlan10.netdev [NetDev] Name=vlan10 Kind=vlan [VLAN] Id=10 deb9-test$ cat /etc/systemd/network/12-vlan10_attach-to-if.network [Match] Name=ens2 [Network] VLAN=vlan10 deb9-test$ cat /etc/systemd/network/16-vlan10_up.network [Match] Name=vlan10 [Network] DHCP=ipv4 IPv6AcceptRA=no LinkLocalAddressing=no
To test if the virtual machine has connection I use:
deb9-test$ journalctl -b --no-hostname -u systemd-networkd.service -- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. -- Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed Sep 23 20:34:05 systemd[1]: Started Network Service. Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1 Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured deb9-test$
4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this:
deb9-test$ journalctl -b --no-hostname -u systemd-networkd.service -- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. -- Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed Sep 23 20:44:59 systemd[1]: Started Network Service. Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured deb9-test$
Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest.
deb9-test$ grep ^ExecStart= /lib/systemd/system/serial-getty@.service ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM
modify to
ExecStart=-/sbin/agetty --autologin yourloginname --keep-baud 115200,38400,9600 %I $TERM
To list all settings of the bridge you can use:
harley$ find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort
oldstyle linux bridge as hub
This works always with the old linux bridge that do not know anything about VLAN. The trick is to set it to a complete transparent state for all connected interfaces like a hub. But you have to know that the bridge will then forward all packets to all interfaces simultanously. You can do it by setting the ageing time to 0.
Disable systemd-networkd and start networking with ifupdown:
harley$ sudo systemctl stop systemd-networkd Warning: Stopping systemd-networkd.service, but it can still be activated by: systemd-networkd.socket harley$ sudo systemctl disable systemd-networkd Removed /etc/systemd/system/multi-user.target.wants/systemd-networkd.service. Removed /etc/systemd/system/sockets.target.wants/systemd-networkd.socket. harley$ sudo ip link set dev br0 down && sudo ip link del dev br0 harley$ sudo systemctl enable networking.service Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable networking harley$ sudo systemctl start networking.service harley$
Setup the bridge and start it:
harley$ cat /etc/network/interfaces auto br0 iface br0 inet manual bridge_ports enp1s0 bridge_ageing 0 bridge_stp off harley$ sudo ifup br0 Waiting for br0 to get ready (MAXWAIT is 32 seconds). harley$
It's all in place now:
harley$ cat /sys/class/net/br0/bridge/ageing_time 0 harley$ cat /sys/class/net/br0/bridge/stp_state 0 harley$ cat /sys/class/net/br0/bridge/vlan_filtering 0
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets.
References
linux bridge as hub
Now I try to setup #oldstyle linux bridge as hub just with systemd-networkd.
Disable networking with ifupdown and start systemd-networkd:
harley$ sudo systemctl stop networking.service harley$ sudo systemctl disable networking.service Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install disable networking harley$ sudo ip link set dev br0 down && sudo ip link del dev br0 harley$ sudo systemctl enable systemd-networkd Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /lib/systemd/system/systemd-networkd.service. Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /lib/systemd/system/systemd-networkd.socket. harley$ sudo systemctl start systemd-networkd harley$
Setup the bridge and start it:
harley$ cat /etc/systemd/network/08-br0.netdev [NetDev] Name=br0 Kind=bridge [Bridge] AgeingTimeSec=0 STP=false harley$ cat /etc/systemd/network/12-br0_add-enp1s0.network [Match] Name=enp1s0 [Network] Bridge=br0 harley$ cat /etc/systemd/network/16-br0_up.network [Match] Name=br0 harley$ sudo ip link set dev br0 down && sudo ip link del dev br0 harley$ sudo systemctl restart systemd-networkd harley$
AgeingTimeSec=0 is not acepted but should:
harley$ cat /sys/class/net/br0/bridge/ageing_time 30000 (means 300 sec) harley$
But I've found a workaround. Useing a number between .01 and .000001 (there are dots) will set ageing_time to 0. So set AgeingTimeSec=.000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we will get:
harley$ cat /sys/class/net/br0/bridge/ageing_time 0 harley$ cat /sys/class/net/br0/bridge/stp_state 0 harley$ cat /sys/class/net/br0/bridge/vlan_filtering 0 harley$
The guest gets now an IP-Address on boot and is connected to VLAN 10.
Hey guys!!! designs and builds specialty lines of lead oxide production equipment, material handling systems, battery related process machinery, parts, and accessories for the battery, pigment, glass, and chemical industries. http://techbasys.com offers technical application and engineering services to help the customer acheive maximum benefit from their equipment and manufacturing processes.
References
- https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
- https://www.freedesktop.org/software/systemd/man/systemd.network.html
linux bridge with libvirt hook scripts
We setup a bridge with VLAN enabled:
harley$ cat /etc/systemd/network/08-br0.netdev [NetDev] Name=br0 Kind=bridge [Bridge] DefaultPVID=none VLANFiltering=true STP=false harley$ cat /etc/systemd/network/12-br0_add-enp1s0.network [Match] Name=enp1s0 [Network] Bridge=br0 [BridgeVLAN] VLAN=10 [BridgeVLAN] VLAN=20 [BridgeVLAN] VLAN=30 harley$ cat /etc/systemd/network/16-br0_up.network [Match] Name=br0
With this I get:
harley$ sudo bridge vlan show port vlan ids enp1s0 1 PVID Egress Untagged 10 20 30 br0 1 PVID Egress Untagged harley$
But what is this? We have default VLAN 1 PVID Egress Untagged
. I don't want this. Seems setting DefaultPVID=none
in 08-br0.netdev doesn't work. I've made a Workaround for setting DefaultPVID=none. Looking at this behavior I found that we can set default_pvid
in the kernel only if vlan_filtering = 0
. By hand I have to do:
harley$ sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering' harley$ sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid' harley$ sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering' harley$
If I start a guest I will get now:
harley$ virsh start deb9-test harley$ sudo bridge vlan show port vlan ids enp1s0 10 20 30 br0 None vnet0 None harley$
The virtual network interface vnet0 for deb9-test has no VLAN ID. Libvirt does not know something about this so we have to tell it. Libvirt provides hook scripts that we can use for this. We have to:
- #define VLAN-ID the virtual machine belongs to
- #get information on startup from the runtime XML-config of the domain
- #set VLAN-ID to the dynamic virtual network interface vnet*
For debugging the hook-scripts I've made a small script:
harley$ cat /etc/libvirt/hooks/debug.sh #!/bin/bash -e # https://www.libvirt.org/hooks.html # If you make a new hook script then 'sudo systemctl restart libvirtd'. # For debug set symlink to hook-script daemon, qemu, lxc, libxl and/or network, # e.g. 'sudo ln -s debug.sh qemu' and restart libvirtd. logfile='/var/log/libvirt/hooks.log' echo "$0" >>$logfile date -Iseconds >>$logfile echo "\$1=$1, \$2=$2, \$3=$3, \$4=$4" >>$logfile cat - >>$logfile echo -e "\n---------------------------------------------" >>$logfile harley$
define VLAN-ID the virtual machine belongs to
For this we have an extra element <metadata> in Domain XML format for custom metadata. We can simply add the information to the static configuration with harley$ virsh edit deb9-test
like this (look only at the <metadata> element):
harley$ virsh dumpxml deb9-test | head -n9 <domain type='kvm' id='1'> <name>deb9-test</name> <uuid>70d56a28-795d-4010-9403-513a4bd6b66a</uuid> <metadata> <my:home xmlns:my="http://hoeft-online.de/my/"> <my:vlan>10</my:vlan> </my:home> </metadata> <memory unit='KiB'>1048576</memory>
get information on startup from the runtime XML-config of the domain
It seems a little bit difficult to get needed information out of the big XML-config but it's no problem with XSLT. I've made a XSL-stylesheet for this and use xmlstarlet. Start a virtual machine and then its runtime configuration is available with harley$ virsh dumpxml deb9-test | xmlstarlet tr qemu.xsl
. With this I can test my stylesheet. Here is it:
harley$ cat /etc/libvirt/hooks/qemu.xsl <?xml version="1.0" encoding="UTF-8"?> <!-- This stylesheet extracts the VLAN-ID and the target device of the bridge from the domain-xml given to the libvirt hook-script "qemu". Example output: <meta><vlan>10</vlan><dev>vnet0</dev></meta> --> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:my="http://hoeft-online.de/my/" exclude-result-prefixes="my"> <xsl:output omit-xml-declaration="yes" indent="no" encoding="utf-8" media-type="text/xml"/> <xsl:strip-space elements="*"/> <xsl:template match="text()|@*"/> <xsl:template match="/domain"> <meta> <xsl:apply-templates/> </meta> </xsl:template> <xsl:template match="metadata/my:home/my:vlan"> <vlan> <xsl:value-of select="."/> </vlan> </xsl:template> <xsl:template match='interface[@type="bridge"]/target'> <dev> <xsl:value-of select="@dev"/> </dev> </xsl:template> </xsl:stylesheet> harley$
harley$ virsh dumpxml deb9-test | xmlstarlet tr /etc/libvirt/hooks/qemu.xsl <meta><vlan>10</vlan><dev>vnet0</dev></meta>harley$
set VLAN-ID to the dynamic virtual network interface vnet*
Putting it all together here is the executable hook-script:
harley$ cat /etc/libvirt/hooks/qemu #!/bin/bash #/etc/libvirt/hooks/qemu # Docs: https://www.libvirt.org/hooks.html # If you make a new hook script then 'sudo systemctl restart libvirtd'. # On startup of the domain (guest) This script does: # Get Metadata VLAN-ID of the guest and target device of the bridge from # the domain-xml available on standard input. It is the runtime # version from 'virsh dumpxml domainname'. For extracting the # information we use a XSL-stylesheet. Example input into $META: # <meta><vlan>10</vlan><dev>vnet0</dev></meta> # Select $DEV from $META # Select $VLAN from $META # Set $VLAN to $DEV on the bridge case "$2" in prepare) ;; start) META=$(/usr/bin/xmlstarlet tr /etc/libvirt/hooks/qemu.xsl -) DEV=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/dev') VLAN=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/vlan') if [[ -n $DEV && -n $VLAN ]]; then /sbin/bridge vlan add vid "$VLAN" dev "$DEV" fi ;; started) ;; stopped) ;; release) ;; migrate) ;; restore) ;; reconnect) ;; attach) ;; *) echo "qemu hook called with unexpected options $*" >&2 exit 1 ;; esac harley$ sudo chmod 744 /etc/libvirt/hooks/qemu harley$
References
- https://www.libvirt.org/hooks.html
- https://serverfault.com/questions/696011/libvirt-hook-qemu-suse12
Workaround for setting DefaultPVID=none
We do not need it anymore. This bug is fixed in systemd 234.
Setting DefaultPVID in asystemd-networkd
configuration file to "none" does not work. Until this bug is fixed I've made a workaround. The kernel accepts setting default_pvid
to 0 (means "none") only if vlan_filtering=0
, so we have to do:
harley$ sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering' harley$ sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid' harley$ sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering' harley$
Check with listing of bridge-settings.
Theese commands must run with systemd-networkd
so we need a service for this. First I make a script and make it executable for root:
harley$ cat /etc/systemd/network/DefaultPVID.sh #!/bin/bash #echo "entering DefaultPVID.sh" >>/tmp/debug.log BRDIR="/sys/class/net/br0/bridge/" if [[ -f $BRDIR/vlan_filtering && -f $BRDIR/default_pvid ]]; then #echo "setting DefaultPVID" >>/tmp/debug.log VLAN_FILTERING="$(cat "$BRDIR"/vlan_filtering)" echo 0 >"$BRDIR"/vlan_filtering echo 0 >"$BRDIR"/default_pvid echo "$VLAN_FILTERING" >"$BRDIR"/vlan_filtering fi exit 0 harley$ sudo chmod 744 /etc/systemd/network/DefaultPVID.sh harley$
Test with harley$ sudo /etc/systemd/network/DefaultPVID.sh
. Next I create a service to execute this script:
harley$ cat /etc/systemd/system/DefaultPVID.service [Unit] Description=set DefaultPVID on a bridge as workaround Wants=network.target After=network.target [Service] Type=oneshot ExecStart=/etc/systemd/network/DefaultPVID.sh [Install] WantedBy=multi-user.target harley$
Test with harley$ sudo systemctl start DefaultPVID.service && systemctl status DefaultPVID.service
. After this I create a drop-in file for overriding vendor settings so this service will be executed together with systemd-networkd
:
harley$ cat /etc/systemd/system/systemd-networkd.service.d/DefaultPVID.conf [Unit] # This is only a workaround. DefaultPVID cannot be set in # /etc/systemd/network/br0.netdev. It seems buggy. Wants=DefaultPVID.service Before=DefaultPVID.service harley$
Test with harley$ sudo systemctl restart systemd-networkd
.