Ingos Wiki - Benutzerbeiträge [de]

VLAN for virtual machines

2018-08-09T19:26:06Z

Ingo: Änderung 28 von 86.62.117.180 (Diskussion) rückgängig gemacht.

== Introduction ==
I wanted to update VLAN connections for virtual
machines to newer technologies and put a question on
[https://unix.stackexchange.com/questions/392758/setup-vlan-on-linux-bridge-for-virtual-machines-with-systemd unix.stackexchange]. But I do not get any answer. It seems there is very
little knowledge for this out there. So I decided to work on it by myself
and document it here.

In gerneral I will look at three methods:
# [[#oldstyle linux bridge as hub|oldstyle linux bridge as hub]]
# [[#linux bridge as hub|linux bridge as hub]]
# [[#linux bridge with libvirt hook scripts|linux bridge with libvirt hook scripts]]

== Preparation ==
I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: [[Setup KVM with console]]. I'm sitting on harley as host, my all day workstation.
Now I start the virtual machine, login and show its interface setting:
'''harley$''' virsh start --console deb9-test
login
'''deb9-test$''' cat /etc/systemd/network/08-vlan10.netdev
[NetDev]
Name=vlan10
Kind=vlan
[VLAN]
Id=10
'''deb9-test$''' cat /etc/systemd/network/12-vlan10_attach-to-if.network
[Match]
Name=ens2
[Network]
VLAN=vlan10
'''deb9-test$''' cat /etc/systemd/network/16-vlan10_up.network
[Match]
Name=vlan10
[Network]
DHCP=ipv4
IPv6AcceptRA=no
LinkLocalAddressing=no
To test if the virtual machine has connection I use:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. --
Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed
Sep 23 20:34:05 systemd[1]: Started Network Service.
Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready
Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success
Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier
Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier
Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL
Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL
Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1
Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured
Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured
'''deb9-test$'''
4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. --
Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed
Sep 23 20:44:59 systemd[1]: Started Network Service.
Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready
Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success
Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier
Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier
Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL
Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL
Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured
'''deb9-test$'''

Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest.
'''deb9-test$''' grep ^ExecStart= /lib/systemd/system/serial-getty@.service
ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM
modify to
ExecStart=-/sbin/agetty --autologin ''yourloginname'' --keep-baud 115200,38400,9600 %I $TERM
To list all settings of the bridge you can use:
'''harley$''' find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort

== oldstyle linux bridge as hub ==
This works always with the old linux bridge that do not know anything about VLAN. The trick is to set it to a complete transparent state for all connected interfaces like a hub. But you have to know that the bridge will then forward all packets to all interfaces simultanously. You can do it by setting the ageing time to 0.

Disable systemd-networkd and start networking with ifupdown:
'''harley$''' sudo systemctl stop systemd-networkd
Warning: Stopping systemd-networkd.service, but it can still be activated by:
systemd-networkd.socket
'''harley$''' sudo systemctl disable systemd-networkd
Removed /etc/systemd/system/multi-user.target.wants/systemd-networkd.service.
Removed /etc/systemd/system/sockets.target.wants/systemd-networkd.socket.
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable networking
'''harley$''' sudo systemctl start networking.service
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/network/interfaces
auto br0
iface br0 inet manual
bridge_ports enp1s0
bridge_ageing 0
bridge_stp off
'''harley$''' sudo ifup br0
Waiting for br0 to get ready (MAXWAIT is 32 seconds).
'''harley$'''
It's all in place now:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets.

=== References ===
* https://wiki.debian.org/NetworkConfiguration#Bridging_without_Switching

== linux bridge as hub ==
Now I try to setup [[#oldstyle linux bridge as hub]] just with systemd-networkd.

Disable networking with ifupdown and start systemd-networkd:
'''harley$''' sudo systemctl stop networking.service
'''harley$''' sudo systemctl disable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable networking
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable systemd-networkd
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /lib/systemd/system/systemd-networkd.socket.
'''harley$''' sudo systemctl start systemd-networkd
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
AgeingTimeSec=0
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl restart systemd-networkd
'''harley$'''

AgeingTimeSec=0 is not acepted but should:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
30000 (means 300 sec)
'''harley$'''
But I've found a workaround. Useing a number between '''.'''01 and '''.'''000001 (there are dots) will set ageing_time to 0.
So set AgeingTimeSec='''.'''000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we
will get:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
'''harley$'''
The guest gets now an IP-Address on boot and is connected to VLAN 10.

=== Discussion ===
This works because of [[#References|three conditions]].
# ageing time is 0: ageing time specifies the number of seconds a MAC Address will be kept in the forwarding database after having a packet received from this MAC Address. Setting it to 0 means there is never a MAC Address stored in the FDB.
# unicast flood on interfaces is on: this controls whether the bridge should flood traffic for which an FDB entry is missing and the destination is unknown through this port. Defaults to on.
# spanning tree protocol (stp) is disabled: we don't have a forward_delay at startup for the learning phase of spanning tree.
I have a running and connected virtual machine:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
br0 1 PVID Egress Untagged
vnet0 1 PVID Egress Untagged
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/forward_delay
1500
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
Indeed we have forward_delay 1500 (means 15 sec) but it doesn't matter. stp_state is 0 (disabled), no spanning tree. Flood (means unicast flood) is on as I can see:
'''harley$''' sudo bridge -d link show
''3: enp1s0'' state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
''95: vnet0'' state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
'''harley$'''

Let's have a look at flooding on the interfaces. I disable it on the physical interface enp1s0 of the bridge and reboot the guest:
'''harley$''' sudo bridge link set dev enp1s0 flood off
'''harley$'''
The guest gets an IP-Address from the DHCP-Server but then can't ping its gateway. DHCP-REQUEST is broadcast and goes thru enp1s0. DHCP-ANSWER comes back thru it to any other (here only vnet0) interface which has flood on. Ping is unicast and isn't forwareded on enp1s0. If I set enp1s0 flood on and vnet0 flood off and <code>'''deb9-test$ '''sudo systemctl restart systemd-networkd</code>, I get no IP-Address from DHCP-Server and can't ping the interface. Incoming DHCP-ANSWER isn't broadcast and vnet0 doesn't forward it to the guest.

Btw. this method has bad performance as we can see with monitor. We insert MAC-Addresses into FDB for just deleting it immediately, all for nothing.
'''harley$''' sudo bridge monitor fdb
52:54:00:01:76:20 dev enp1s0 master br0
52:54:00:b0:ca:63 dev vnet0 master br0
f4:f2:6d:2c:87:f7 dev enp1s0 master br0
00:80:3f:2a:31:1a dev enp1s0 master br0
Deleted 52:54:00:01:76:20 dev enp1s0 master br0 stale
Deleted 52:54:00:b0:ca:63 dev vnet0 master br0 stale
Deleted 00:80:3f:2a:31:1a dev enp1s0 master br0 stale
Deleted f4:f2:6d:2c:87:f7 dev enp1s0 master br0 stale
...

=== References ===
* https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
* https://www.freedesktop.org/software/systemd/man/systemd.network.html

== linux bridge with libvirt hook scripts ==
We setup a bridge with VLAN enabled:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
DefaultPVID=none
VLANFiltering=true
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
[BridgeVLAN]
VLAN=10
[BridgeVLAN]
VLAN=20
[BridgeVLAN]
VLAN=30
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
With this I get:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
10
20
30
br0 1 PVID Egress Untagged
'''harley$'''
But what is this? We have default VLAN <code>1 PVID Egress Untagged</code>. I don't want this. Seems setting <code>DefaultPVID=none</code> in 08-br0.netdev doesn't work. I've made a [[#Workaround for setting DefaultPVID=none|Workaround for setting DefaultPVID=none]]. Looking at this behavior I found that we can set <code>default_pvid</code> in the kernel only if <code>vlan_filtering = 0</code>. By hand I have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
If I start a guest I will get now:
'''harley$''' virsh start deb9-test
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 10
20
30
br0 None
vnet0 None
'''harley$'''
The virtual network interface vnet0 for deb9-test has no VLAN ID. Libvirt does not know something about this so we have to tell it. Libvirt provides [https://www.libvirt.org/hooks.html hook scripts] that we can use for this. We have to:
# [[#define VLAN-ID the virtual machine belongs to]]
# [[#get information on startup from the runtime XML-config of the domain]]
# [[#set VLAN-ID to the dynamic virtual network interface vnet*]]
For debugging the hook-scripts I've made a small script:
'''harley$''' cat /etc/libvirt/hooks/debug.sh
#!/bin/bash -e
# https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.
# For debug set symlink to hook-script daemon, qemu, lxc, libxl and/or network,
# e.g. 'sudo ln -s debug.sh qemu' and restart libvirtd.

logfile='/var/log/libvirt/hooks.log'

echo "$0" >>$logfile
date -Iseconds >>$logfile
echo "\$1=$1, \$2=$2, \$3=$3, \$4=$4" >>$logfile
cat - >>$logfile
echo -e "\n---------------------------------------------" >>$logfile
'''harley$'''

=== define VLAN-ID the virtual machine belongs to ===
For this we have an extra [https://libvirt.org/formatdomain.html#elementsMetadata element <metadata> in Domain XML format] for custom metadata. We can simply add the information to the static configuration with <code>'''harley$''' virsh edit deb9-test</code> like this (look only at the <metadata> element):
'''harley$''' virsh dumpxml deb9-test | head -n9
<domain type='kvm' id='1'>
<name>deb9-test</name>
<uuid>70d56a28-795d-4010-9403-513a4bd6b66a</uuid>
<metadata>
<my:home xmlns:my="http://hoeft-online.de/my/">
<my:vlan>10</my:vlan>
</my:home>
</metadata>
<memory unit='KiB'>1048576</memory>

=== get information on startup from the runtime XML-config of the domain ===
It seems a little bit difficult to get needed information out of the big XML-config but it's no problem with XSLT. I've made a XSL-stylesheet for this and use xmlstarlet. Start a virtual machine and then its runtime configuration is available with <code>'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr qemu.xsl</code>. With this I can test my stylesheet. Here is it:
'''harley$''' cat /etc/libvirt/hooks/qemu.xsl
<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:my="http://hoeft-online.de/my/" exclude-result-prefixes="my">
<xsl:output omit-xml-declaration="yes" indent="no"
encoding="utf-8" media-type="text/xml"/>
<xsl:strip-space elements="*"/>
<xsl:template match="text()|@*"/>

<xsl:template match="/domain">
<meta>
<xsl:apply-templates/>
</meta>
</xsl:template>

<xsl:template match="metadata/my:home/my:vlan">
<vlan>
<xsl:value-of select="."/>
</vlan>
</xsl:template>

<xsl:template match='interface[@type="bridge"]/target'>
<dev>
<xsl:value-of select="@dev"/>
</dev>
</xsl:template>

</xsl:stylesheet>
'''harley$'''

'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr /etc/libvirt/hooks/qemu.xsl
<meta><vlan>10</vlan><dev>vnet0</dev></meta>'''harley$'''

=== set VLAN-ID to the dynamic virtual network interface vnet* ===
Putting it all together here is the executable hook-script:
'''harley$''' cat /etc/libvirt/hooks/qemu
#!/bin/bash
#/etc/libvirt/hooks/qemu
# Docs: https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.

# On startup of the domain (guest) This script does:
# Get Metadata VLAN-ID of the guest and target device of the bridge from
# the domain-xml available on standard input. It is the runtime
# version from 'virsh dumpxml domainname'. For extracting the
# information we use a XSL-stylesheet. Example input into $META:
# <meta><vlan>10</vlan><dev>vnet0</dev></meta>
# Select $DEV from $META
# Select $VLAN from $META
# Set $VLAN to $DEV on the bridge

case "$2" in
prepare)
;;
start)
META=$(/usr/bin/xmlstarlet tr /etc/libvirt/hooks/qemu.xsl -)
DEV=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/dev')
VLAN=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/vlan')
if [[ -n $DEV && -n $VLAN ]]; then
/sbin/bridge vlan add vid "$VLAN" dev "$DEV"
fi
;;
started)
;;
stopped)
;;
release)
;;
migrate)
;;
restore)
;;
reconnect)
;;
attach)
;;
*)
echo "qemu hook called with unexpected options $*" >&2
exit 1
;;
esac
'''harley$''' sudo chmod 744 /etc/libvirt/hooks/qemu
'''harley$'''

=== References ===
* https://www.libvirt.org/hooks.html
* https://serverfault.com/questions/696011/libvirt-hook-qemu-suse12

== Workaround for setting DefaultPVID=none ==
We do not need it anymore. This bug is fixed in systemd 234.

Setting [https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#DefaultPVID= DefaultPVID] in a<code>systemd-networkd</code> configuration file to "none" does not work. Until this bug is fixed I've made a workaround. The kernel accepts setting <code>default_pvid</code> to 0 (means "none") only if <code>vlan_filtering=0</code>, so we have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
Check with listing of [[#bridge-settings|bridge-settings]].
Theese commands must run with <code>systemd-networkd</code> so we need a service for this. First I make a script and make it executable for root:
'''harley$''' cat /etc/systemd/network/DefaultPVID.sh
#!/bin/bash
#echo "entering DefaultPVID.sh" >>/tmp/debug.log

BRDIR="/sys/class/net/br0/bridge/"

if [[ -f $BRDIR/vlan_filtering && -f $BRDIR/default_pvid ]]; then
#echo "setting DefaultPVID" >>/tmp/debug.log
VLAN_FILTERING="$(cat "$BRDIR"/vlan_filtering)"
echo 0 >"$BRDIR"/vlan_filtering
echo 0 >"$BRDIR"/default_pvid
echo "$VLAN_FILTERING" >"$BRDIR"/vlan_filtering
fi
exit 0
'''harley$''' sudo chmod 744 /etc/systemd/network/DefaultPVID.sh
'''harley$'''
Test with <code>'''harley$''' sudo /etc/systemd/network/DefaultPVID.sh</code>. Next I create a service to execute this script:
'''harley$''' cat /etc/systemd/system/DefaultPVID.service
[Unit]
Description=set DefaultPVID on a bridge as workaround
Wants=network.target
After=network.target

[Service]
Type=oneshot
ExecStart=/etc/systemd/network/DefaultPVID.sh

[Install]
WantedBy=multi-user.target
'''harley$'''
Test with <code>'''harley$''' sudo systemctl start DefaultPVID.service && systemctl status DefaultPVID.service</code>. After this I create a [https://www.freedesktop.org/software/systemd/man/systemd.unit.html#id-1.11.3 drop-in file for overriding vendor settings] so this service will be executed together with <code>systemd-networkd</code>:
'''harley$''' cat /etc/systemd/system/systemd-networkd.service.d/DefaultPVID.conf
[Unit]
# This is only a workaround. DefaultPVID cannot be set in
# /etc/systemd/network/br0.netdev. It seems buggy.
Wants=DefaultPVID.service
Before=DefaultPVID.service
'''harley$'''
Test with <code>'''harley$''' sudo systemctl restart systemd-networkd</code>.

[[Category:Virtualization]]

Raspberry Pi as UPnP renderer

2017-10-25T14:25:04Z

Ingo: add description for starting gmediarender

Date: 2017-10-25 
lsb_release Description: Raspbian GNU/Linux 9.1 (stretch)

I setup a multiroom music playing environment. For this I use Raspberry Pis for renderer in a room. Here I found a [https://joachim-wilke.de/blog/2016/07/10/UPNP-Renderer-auf-dem-Raspberry-Pi/ somewhat outdated howto] but it points me in the right direction. Here is howto install it.

At least the default audio output should do. You can download this small file [http://hoeft-online.de/working.wav working.wav] and test it with
'''pi@raspberrypi:~ $''' aplay working.wav
'''pi@raspberrypi:~ $'''

We use [https://github.com/hzeller/gmrender-resurrect gmrender-resurrect] which is available in the repository. Install it on a RasPi with needed plugins.

'''pi@raspberrypi:~ $''' sudo apt install gmediarender gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-alsa
'''pi@raspberrypi:~ $'''

Test installation with
'''pi@raspberrypi:~ $''' /usr/bin/gmediarender -f Test -u 42 --logfile /dev/stdout

Configure in <code>/etc/default/gmediarender</code>, set <code>ENABLED=1</code> and maybe your <code>UPNP_DEVICE_NAME</code>. Don't use "special" characters like umlaut. Startup will quit with an error message. Seems gmediarender still not understand UTF-8 :-(

Start gmediarender as service:
'''pi@raspberrypi:~ $''' sudo systemctl enable gmediarender.service
'''pi@raspberrypi:~ $''' sudo systemctl start gmediarender.service
'''pi@raspberrypi:~ $''' sudo systemctl status gmediarender.service

The output of the last command should contain a line like: 
<code>   Active: active (running) since Wed 2017-10-25 14:17:09 UTC; 2s ago</code>
[[Category:Raspberry Pi]]

Raspberry Pi as UPnP renderer

2017-10-25T10:57:07Z

Ingo: add audio testfile

Date: 2017-10-24 
lsb_release Description: Raspbian GNU/Linux 9.1 (stretch)

I setup a multiroom music playing environment. For this I use Raspberry Pis for renderer in a room. Here I found a [https://joachim-wilke.de/blog/2016/07/10/UPNP-Renderer-auf-dem-Raspberry-Pi/ somewhat outdated howto] but it points me in the right direction. Here is howto install it.

At least the default audio output should do. You can download this small file [http://hoeft-online.de/working.wav working.wav] and test it with
'''pi@raspberrypi:~ $''' aplay working.wav
'''pi@raspberrypi:~ $'''

We use [https://github.com/hzeller/gmrender-resurrect gmrender-resurrect] which is available in the repository. Install it on a RasPi with needed plugins.

'''pi@raspberrypi:~ $''' sudo apt install gmediarender gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-alsa
'''pi@raspberrypi:~ $'''

Test installation with
'''pi@raspberrypi:~ $''' /usr/bin/gmediarender -f Test -u 42 --logfile /dev/stdout

Configure in /etc/default/gmediarender. Don't use "special" characters like umlaut. Startup will quit with an error message. Seems gmediarender still not understand UTF-8 :-(

Start gmediarender as service:
'''pi@raspberrypi:~ $''' sudo systemctl enable gmediarender.service
'''pi@raspberrypi:~ $''' sudo systemctl start gmediarender.service
'''pi@raspberrypi:~ $'''

[[Category:Raspberry Pi]]

Raspberry Pi as UPnP renderer

2017-10-25T01:29:03Z

Ingo: software version added

Date: 2017-10-24 
lsb_release Description: Raspbian GNU/Linux 9.1 (stretch)

I setup a multiroom music playing environment. For this I use Raspberry Pis for renderer in a room. Here I found a [https://joachim-wilke.de/blog/2016/07/10/UPNP-Renderer-auf-dem-Raspberry-Pi/ somewhat outdated howto] but it points me in the right direction. Here is howto install it.

At least the default audio output should do. Test it with

We use [https://github.com/hzeller/gmrender-resurrect gmrender-resurrect] which is available in the repository. Install it on a RasPi with needed plugins.

'''pi@raspberrypi:~ $''' sudo apt install gmediarender gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-alsa
'''pi@raspberrypi:~ $'''

Test installation with
'''pi@raspberrypi:~ $''' /usr/bin/gmediarender -f Test -u 42 --logfile /dev/stdout

Configure in /etc/default/gmediarender. Don't use "special" characters like umlaut. Startup will quit with an error message. Seems gmediarender still not understand UTF-8 :-(

Start gmediarender as service:
'''pi@raspberrypi:~ $''' sudo systemctl enable gmediarender.service
'''pi@raspberrypi:~ $''' sudo systemctl start gmediarender.service
'''pi@raspberrypi:~ $'''

[[Category:Raspberry Pi]]

Kategorie:Raspberry Pi

2017-10-24T23:26:57Z

Ingo: create page

Everything to Raspberry Pi

Raspberry Pi as UPnP renderer

2017-10-24T22:32:23Z

Ingo: create page

Date: 2017-10-24

I setup a multiroom music playing environment. For this I use Raspberry Pis for renderer in a room. Here I found a [https://joachim-wilke.de/blog/2016/07/10/UPNP-Renderer-auf-dem-Raspberry-Pi/ somewhat outdated howto] but it points me in the right direction. Here is howto install it.

We use [https://github.com/hzeller/gmrender-resurrect gmrender-resurrect] which is available in the repository. Install it on a RasPi with needed plugins.

'''pi@raspberrypi:~ $''' sudo apt install gmediarender gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-alsa
'''pi@raspberrypi:~ $'''

Test installation with
'''pi@raspberrypi:~ $''' /usr/bin/gmediarender -f Test -u 42 --logfile /dev/stdout

Configure in /etc/default/gmediasrender. Don't use "special" characters like umlaut. Startup will quit with an error message. Seems gmediarender still not understand UTF-8 :-(

Start gmediarender as service:
'''pi@raspberrypi:~ $''' sudo systemctl enable gmediarender.service
'''pi@raspberrypi:~ $''' sudo systemctl start gmediarender.service
'''pi@raspberrypi:~ $'''

[[Category:Raspberry Pi]]

VLAN for virtual machines

2017-09-28T15:41:56Z

Ingo: bug is fixed

VLAN for virtual machines

2017-09-28T08:23:49Z

Ingo: state Workaround for DefaultPVID more precisely

== Introduction ==
I wanted to update VLAN connections for virtual
machines to newer technologies and put a question on
[https://unix.stackexchange.com/questions/392758/setup-vlan-on-linux-bridge-for-virtual-machines-with-systemd unix.stackexchange]. But I do not get any answer. It seems there is very
little knowledge for this out there. So I decided to work on it by myself
and document it here.

In gerneral I will look at three methods:
# [[#oldstyle linux bridge as hub|oldstyle linux bridge as hub]]
# [[#linux bridge as hub|linux bridge as hub]]
# [[#linux bridge with libvirt hook scripts|linux bridge with libvirt hook scripts]]

== Preparation ==
I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: [[Setup KVM with console]]. I'm sitting on harley as host, my all day workstation.
Now I start the virtual machine, login and show its interface setting:
'''harley$''' virsh start --console deb9-test
login
'''deb9-test$''' cat /etc/systemd/network/08-vlan10.netdev
[NetDev]
Name=vlan10
Kind=vlan
[VLAN]
Id=10
'''deb9-test$''' cat /etc/systemd/network/12-vlan10_attach-to-if.network
[Match]
Name=ens2
[Network]
VLAN=vlan10
'''deb9-test$''' cat /etc/systemd/network/16-vlan10_up.network
[Match]
Name=vlan10
[Network]
DHCP=ipv4
IPv6AcceptRA=no
LinkLocalAddressing=no
To test if the virtual machine has connection I use:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. --
Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed
Sep 23 20:34:05 systemd[1]: Started Network Service.
Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready
Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success
Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier
Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier
Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL
Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL
Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1
Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured
Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured
'''deb9-test$'''
4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. --
Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed
Sep 23 20:44:59 systemd[1]: Started Network Service.
Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready
Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success
Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier
Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier
Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL
Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL
Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured
'''deb9-test$'''

Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest.
'''deb9-test$''' grep ^ExecStart= /lib/systemd/system/serial-getty@.service
ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM
modify to
ExecStart=-/sbin/agetty --autologin ''yourloginname'' --keep-baud 115200,38400,9600 %I $TERM
To list all settings of the bridge you can use:
'''harley$''' find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort

== oldstyle linux bridge as hub ==
This works always with the old linux bridge that do not know anything about VLAN. The trick is to set it to a complete transparent state for all connected interfaces like a hub. But you have to know that the bridge will then forward all packets to all interfaces simultanously. You can do it by setting the ageing time to 0.

Disable systemd-networkd and start networking with ifupdown:
'''harley$''' sudo systemctl stop systemd-networkd
Warning: Stopping systemd-networkd.service, but it can still be activated by:
systemd-networkd.socket
'''harley$''' sudo systemctl disable systemd-networkd
Removed /etc/systemd/system/multi-user.target.wants/systemd-networkd.service.
Removed /etc/systemd/system/sockets.target.wants/systemd-networkd.socket.
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable networking
'''harley$''' sudo systemctl start networking.service
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/network/interfaces
auto br0
iface br0 inet manual
bridge_ports enp1s0
bridge_ageing 0
bridge_stp off
'''harley$''' sudo ifup br0
Waiting for br0 to get ready (MAXWAIT is 32 seconds).
'''harley$'''
It's all in place now:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets.

=== References ===
* https://wiki.debian.org/NetworkConfiguration#Bridging_without_Switching

== linux bridge as hub ==
Now I try to setup [[#oldstyle linux bridge as hub]] just with systemd-networkd.

Disable networking with ifupdown and start systemd-networkd:
'''harley$''' sudo systemctl stop networking.service
'''harley$''' sudo systemctl disable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable networking
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable systemd-networkd
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /lib/systemd/system/systemd-networkd.socket.
'''harley$''' sudo systemctl start systemd-networkd
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
AgeingTimeSec=0
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl restart systemd-networkd
'''harley$'''

AgeingTimeSec=0 is not acepted but should:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
30000 (means 300 sec)
'''harley$'''
But I've found a workaround. Useing a number between '''.'''01 and '''.'''000001 (there are dots) will set ageing_time to 0.
So set AgeingTimeSec='''.'''000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we
will get:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
'''harley$'''
The guest gets now an IP-Address on boot and is connected to VLAN 10.

=== Discussion ===
This works because of [[#References|three conditions]].
# ageing time is 0: ageing time specifies the number of seconds a MAC Address will be kept in the forwarding database after having a packet received from this MAC Address. Setting it to 0 means there is never a MAC Address stored in the FDB.
# unicast flood on interfaces is on: this controls whether the bridge should flood traffic for which an FDB entry is missing and the destination is unknown through this port. Defaults to on.
# spanning tree protocol (stp) is disabled: we don't have a forward_delay at startup for the learning phase of spanning tree.
I have a running and connected virtual machine:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
br0 1 PVID Egress Untagged
vnet0 1 PVID Egress Untagged
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/forward_delay
1500
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
Indeed we have forward_delay 1500 (means 15 sec) but it doesn't matter. stp_state is 0 (disabled), no spanning tree. Flood (means unicast flood) is on as I can see:
'''harley$''' sudo bridge -d link show
''3: enp1s0'' state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
''95: vnet0'' state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
'''harley$'''

Let's have a look at flooding on the interfaces. I disable it on the physical interface enp1s0 of the bridge and reboot the guest:
'''harley$''' sudo bridge link set dev enp1s0 flood off
'''harley$'''
The guest gets an IP-Address from the DHCP-Server but then can't ping its gateway. DHCP-REQUEST is broadcast and goes thru enp1s0. DHCP-ANSWER comes back thru it to any other (here only vnet0) interface which has flood on. Ping is unicast and isn't forwareded on enp1s0. If I set enp1s0 flood on and vnet0 flood off and <code>'''deb9-test$ '''sudo systemctl restart systemd-networkd</code>, I get no IP-Address from DHCP-Server and can't ping the interface. Incoming DHCP-ANSWER isn't broadcast and vnet0 doesn't forward it to the guest.

Btw. this method has bad performance as we can see with monitor. We insert MAC-Addresses into FDB for just deleting it immediately, all for nothing.
'''harley$''' sudo bridge monitor fdb
52:54:00:01:76:20 dev enp1s0 master br0
52:54:00:b0:ca:63 dev vnet0 master br0
f4:f2:6d:2c:87:f7 dev enp1s0 master br0
00:80:3f:2a:31:1a dev enp1s0 master br0
Deleted 52:54:00:01:76:20 dev enp1s0 master br0 stale
Deleted 52:54:00:b0:ca:63 dev vnet0 master br0 stale
Deleted 00:80:3f:2a:31:1a dev enp1s0 master br0 stale
Deleted f4:f2:6d:2c:87:f7 dev enp1s0 master br0 stale
...

=== References ===
* https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
* https://www.freedesktop.org/software/systemd/man/systemd.network.html

== linux bridge with libvirt hook scripts ==
We setup a bridge with VLAN enabled:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
DefaultPVID=none
VLANFiltering=true
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
[BridgeVLAN]
VLAN=10
[BridgeVLAN]
VLAN=20
[BridgeVLAN]
VLAN=30
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
With this I get:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
10
20
30
br0 1 PVID Egress Untagged
'''harley$'''
But what is this? We have default VLAN <code>1 PVID Egress Untagged</code>. I don't want this. Seems setting <code>DefaultPVID=none</code> in 08-br0.netdev doesn't work. I've made a [[#Workaround for setting DefaultPVID=none|Workaround for setting DefaultPVID=none]]. Looking at this behavior I found that we can set <code>default_pvid</code> in the kernel only if <code>vlan_filtering = 0</code>. By hand I have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
If I start a guest I will get now:
'''harley$''' virsh start deb9-test
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 10
20
30
br0 None
vnet0 None
'''harley$'''
The virtual network interface vnet0 for deb9-test has no VLAN ID. Libvirt does not know something about this so we have to tell it. Libvirt provides [https://www.libvirt.org/hooks.html hook scripts] that we can use for this. We have to:
# [[#define VLAN-ID the virtual machine belongs to]]
# [[#get information on startup from the runtime XML-config of the domain]]
# [[#set VLAN-ID to the dynamic virtual network interface vnet*]]
For debugging the hook-scripts I've made a small script:
'''harley$''' cat /etc/libvirt/hooks/debug.sh
#!/bin/bash -e
# https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.
# For debug set symlink to hook-script daemon, qemu, lxc, libxl and/or network,
# e.g. 'sudo ln -s debug.sh qemu' and restart libvirtd.

logfile='/var/log/libvirt/hooks.log'

echo "$0" >>$logfile
date -Iseconds >>$logfile
echo "\$1=$1, \$2=$2, \$3=$3, \$4=$4" >>$logfile
cat - >>$logfile
echo -e "\n---------------------------------------------" >>$logfile
'''harley$'''

=== define VLAN-ID the virtual machine belongs to ===
For this we have an extra [https://libvirt.org/formatdomain.html#elementsMetadata element <metadata> in Domain XML format] for custom metadata. We can simply add the information to the static configuration with <code>'''harley$''' virsh edit deb9-test</code> like this (look only at the <metadata> element):
'''harley$''' virsh dumpxml deb9-test | head -n9
<domain type='kvm' id='1'>
<name>deb9-test</name>
<uuid>70d56a28-795d-4010-9403-513a4bd6b66a</uuid>
<metadata>
<my:home xmlns:my="http://hoeft-online.de/my/">
<my:vlan>10</my:vlan>
</my:home>
</metadata>
<memory unit='KiB'>1048576</memory>

=== get information on startup from the runtime XML-config of the domain ===
It seems a little bit difficult to get needed information out of the big XML-config but it's no problem with XSLT. I've made a XSL-stylesheet for this and use xmlstarlet. Start a virtual machine and then its runtime configuration is available with <code>'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr qemu.xsl</code>. With this I can test my stylesheet. Here is it:
'''harley$''' cat /etc/libvirt/hooks/qemu.xsl
<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:my="http://hoeft-online.de/my/" exclude-result-prefixes="my">
<xsl:output omit-xml-declaration="yes" indent="no"
encoding="utf-8" media-type="text/xml"/>
<xsl:strip-space elements="*"/>
<xsl:template match="text()|@*"/>

<xsl:template match="/domain">
<meta>
<xsl:apply-templates/>
</meta>
</xsl:template>

<xsl:template match="metadata/my:home/my:vlan">
<vlan>
<xsl:value-of select="."/>
</vlan>
</xsl:template>

<xsl:template match='interface[@type="bridge"]/target'>
<dev>
<xsl:value-of select="@dev"/>
</dev>
</xsl:template>

</xsl:stylesheet>
'''harley$'''

'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr /etc/libvirt/hooks/qemu.xsl
<meta><vlan>10</vlan><dev>vnet0</dev></meta>'''harley$'''

=== set VLAN-ID to the dynamic virtual network interface vnet* ===
Putting it all together here is the executable hook-script:
'''harley$''' cat /etc/libvirt/hooks/qemu
#!/bin/bash
#/etc/libvirt/hooks/qemu
# Docs: https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.

# On startup of the domain (guest) This script does:
# Get Metadata VLAN-ID of the guest and target device of the bridge from
# the domain-xml available on standard input. It is the runtime
# version from 'virsh dumpxml domainname'. For extracting the
# information we use a XSL-stylesheet. Example input into $META:
# <meta><vlan>10</vlan><dev>vnet0</dev></meta>
# Select $DEV from $META
# Select $VLAN from $META
# Set $VLAN to $DEV on the bridge

case "$2" in
prepare)
;;
start)
META=$(/usr/bin/xmlstarlet tr /etc/libvirt/hooks/qemu.xsl -)
DEV=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/dev')
VLAN=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/vlan')
if [[ -n $DEV && -n $VLAN ]]; then
/sbin/bridge vlan add vid "$VLAN" dev "$DEV"
fi
;;
started)
;;
stopped)
;;
release)
;;
migrate)
;;
restore)
;;
reconnect)
;;
attach)
;;
*)
echo "qemu hook called with unexpected options $*" >&2
exit 1
;;
esac
'''harley$''' sudo chmod 744 /etc/libvirt/hooks/qemu
'''harley$'''

=== References ===
* https://www.libvirt.org/hooks.html
* https://serverfault.com/questions/696011/libvirt-hook-qemu-suse12

== Workaround for setting DefaultPVID=none ==
Setting [https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#DefaultPVID= DefaultPVID] in a<code>systemd-networkd</code> configuration file to "none" does not work. Until this bug is fixed I've made a workaround. The kernel accepts setting <code>default_pvid</code> to 0 (means "none") only if <code>vlan_filtering=0</code>, so we have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
Check with listing of [[#bridge-settings|bridge-settings]].
Theese commands must run with <code>systemd-networkd</code> so we need a service for this. First I make a script and make it executable for root:
'''harley$''' cat /etc/systemd/network/DefaultPVID.sh
#!/bin/bash
#echo "entering DefaultPVID.sh" >>/tmp/debug.log

BRDIR="/sys/class/net/br0/bridge/"

if [[ -f $BRDIR/vlan_filtering && -f $BRDIR/default_pvid ]]; then
#echo "setting DefaultPVID" >>/tmp/debug.log
VLAN_FILTERING="$(cat "$BRDIR"/vlan_filtering)"
echo 0 >"$BRDIR"/vlan_filtering
echo 0 >"$BRDIR"/default_pvid
echo "$VLAN_FILTERING" >"$BRDIR"/vlan_filtering
fi
exit 0
'''harley$''' sudo chmod 744 /etc/systemd/network/DefaultPVID.sh
'''harley$'''
Test with <code>'''harley$''' sudo /etc/systemd/network/DefaultPVID.sh</code>. Next I create a service to execute this script:
'''harley$''' cat /etc/systemd/system/DefaultPVID.service
[Unit]
Description=set DefaultPVID on a bridge as workaround
Wants=network.target
After=network.target

[Service]
Type=oneshot
ExecStart=/etc/systemd/network/DefaultPVID.sh

[Install]
WantedBy=multi-user.target
'''harley$'''
Test with <code>'''harley$''' sudo systemctl start DefaultPVID.service && systemctl status DefaultPVID.service</code>. After this I create a [https://www.freedesktop.org/software/systemd/man/systemd.unit.html#id-1.11.3 drop-in file for overriding vendor settings] so this service will be executed together with <code>systemd-networkd</code>:
'''harley$''' cat /etc/systemd/system/systemd-networkd.service.d/DefaultPVID.conf
[Unit]
# This is only a workaround. DefaultPVID cannot be set in
# /etc/systemd/network/br0.netdev. It seems buggy.
Wants=DefaultPVID.service
Before=DefaultPVID.service
'''harley$'''
Test with <code>'''harley$''' sudo systemctl restart systemd-networkd</code>.

[[Category:Virtualization]]

VLAN for virtual machines

2017-09-27T22:20:48Z

Ingo: some minor corrections

== Introduction ==
I wanted to update VLAN connections for virtual
machines to newer technologies and put a question on
[https://unix.stackexchange.com/questions/392758/setup-vlan-on-linux-bridge-for-virtual-machines-with-systemd unix.stackexchange]. But I do not get any answer. It seems there is very
little knowledge for this out there. So I decided to work on it by myself
and document it here.

In gerneral I will look at three methods:
# [[#oldstyle linux bridge as hub|oldstyle linux bridge as hub]]
# [[#linux bridge as hub|linux bridge as hub]]
# [[#linux bridge with libvirt hook scripts|linux bridge with libvirt hook scripts]]

== Preparation ==
I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: [[Setup KVM with console]]. I'm sitting on harley as host, my all day workstation.
Now I start the virtual machine, login and show its interface setting:
'''harley$''' virsh start --console deb9-test
login
'''deb9-test$''' cat /etc/systemd/network/08-vlan10.netdev
[NetDev]
Name=vlan10
Kind=vlan
[VLAN]
Id=10
'''deb9-test$''' cat /etc/systemd/network/12-vlan10_attach-to-if.network
[Match]
Name=ens2
[Network]
VLAN=vlan10
'''deb9-test$''' cat /etc/systemd/network/16-vlan10_up.network
[Match]
Name=vlan10
[Network]
DHCP=ipv4
IPv6AcceptRA=no
LinkLocalAddressing=no
To test if the virtual machine has connection I use:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. --
Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed
Sep 23 20:34:05 systemd[1]: Started Network Service.
Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready
Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success
Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier
Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier
Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL
Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL
Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1
Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured
Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured
'''deb9-test$'''
4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. --
Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed
Sep 23 20:44:59 systemd[1]: Started Network Service.
Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready
Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success
Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier
Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier
Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL
Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL
Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured
'''deb9-test$'''

Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest.
'''deb9-test$''' grep ^ExecStart= /lib/systemd/system/serial-getty@.service
ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM
modify to
ExecStart=-/sbin/agetty --autologin ''yourloginname'' --keep-baud 115200,38400,9600 %I $TERM
To list all settings of the bridge you can use:
'''harley$''' find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort

== oldstyle linux bridge as hub ==
This works always with the old linux bridge that do not know anything about VLAN. The trick is to set it to a complete transparent state for all connected interfaces like a hub. But you have to know that the bridge will then forward all packets to all interfaces simultanously. You can do it by setting the ageing time to 0.

Disable systemd-networkd and start networking with ifupdown:
'''harley$''' sudo systemctl stop systemd-networkd
Warning: Stopping systemd-networkd.service, but it can still be activated by:
systemd-networkd.socket
'''harley$''' sudo systemctl disable systemd-networkd
Removed /etc/systemd/system/multi-user.target.wants/systemd-networkd.service.
Removed /etc/systemd/system/sockets.target.wants/systemd-networkd.socket.
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable networking
'''harley$''' sudo systemctl start networking.service
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/network/interfaces
auto br0
iface br0 inet manual
bridge_ports enp1s0
bridge_ageing 0
bridge_stp off
'''harley$''' sudo ifup br0
Waiting for br0 to get ready (MAXWAIT is 32 seconds).
'''harley$'''
It's all in place now:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets.

=== References ===
* https://wiki.debian.org/NetworkConfiguration#Bridging_without_Switching

== linux bridge as hub ==
Now I try to setup [[#oldstyle linux bridge as hub]] just with systemd-networkd.

Disable networking with ifupdown and start systemd-networkd:
'''harley$''' sudo systemctl stop networking.service
'''harley$''' sudo systemctl disable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable networking
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable systemd-networkd
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /lib/systemd/system/systemd-networkd.socket.
'''harley$''' sudo systemctl start systemd-networkd
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
AgeingTimeSec=0
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl restart systemd-networkd
'''harley$'''

AgeingTimeSec=0 is not acepted but should:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
30000 (means 300 sec)
'''harley$'''
But I've found a workaround. Useing a number between '''.'''01 and '''.'''000001 (there are dots) will set ageing_time to 0.
So set AgeingTimeSec='''.'''000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we
will get:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
'''harley$'''
The guest gets now an IP-Address on boot and is connected to VLAN 10.

=== Discussion ===
This works because of [[#References|three conditions]].
# ageing time is 0: ageing time specifies the number of seconds a MAC Address will be kept in the forwarding database after having a packet received from this MAC Address. Setting it to 0 means there is never a MAC Address stored in the FDB.
# unicast flood on interfaces is on: this controls whether the bridge should flood traffic for which an FDB entry is missing and the destination is unknown through this port. Defaults to on.
# spanning tree protocol (stp) is disabled: we don't have a forward_delay at startup for the learning phase of spanning tree.
I have a running and connected virtual machine:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
br0 1 PVID Egress Untagged
vnet0 1 PVID Egress Untagged
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/forward_delay
1500
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
Indeed we have forward_delay 1500 (means 15 sec) but it doesn't matter. stp_state is 0 (disabled), no spanning tree. Flood (means unicast flood) is on as I can see:
'''harley$''' sudo bridge -d link show
''3: enp1s0'' state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
''95: vnet0'' state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
'''harley$'''

Let's have a look at flooding on the interfaces. I disable it on the physical interface enp1s0 of the bridge and reboot the guest:
'''harley$''' sudo bridge link set dev enp1s0 flood off
'''harley$'''
The guest gets an IP-Address from the DHCP-Server but then can't ping its gateway. DHCP-REQUEST is broadcast and goes thru enp1s0. DHCP-ANSWER comes back thru it to any other (here only vnet0) interface which has flood on. Ping is unicast and isn't forwareded on enp1s0. If I set enp1s0 flood on and vnet0 flood off and <code>'''deb9-test$ '''sudo systemctl restart systemd-networkd</code>, I get no IP-Address from DHCP-Server and can't ping the interface. Incoming DHCP-ANSWER isn't broadcast and vnet0 doesn't forward it to the guest.

Btw. this method has bad performance as we can see with monitor. We insert MAC-Addresses into FDB for just deleting it immediately, all for nothing.
'''harley$''' sudo bridge monitor fdb
52:54:00:01:76:20 dev enp1s0 master br0
52:54:00:b0:ca:63 dev vnet0 master br0
f4:f2:6d:2c:87:f7 dev enp1s0 master br0
00:80:3f:2a:31:1a dev enp1s0 master br0
Deleted 52:54:00:01:76:20 dev enp1s0 master br0 stale
Deleted 52:54:00:b0:ca:63 dev vnet0 master br0 stale
Deleted 00:80:3f:2a:31:1a dev enp1s0 master br0 stale
Deleted f4:f2:6d:2c:87:f7 dev enp1s0 master br0 stale
...

=== References ===
* https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
* https://www.freedesktop.org/software/systemd/man/systemd.network.html

== linux bridge with libvirt hook scripts ==
We setup a bridge with VLAN enabled:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
DefaultPVID=none
VLANFiltering=true
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
[BridgeVLAN]
VLAN=10
[BridgeVLAN]
VLAN=20
[BridgeVLAN]
VLAN=30
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
With this I get:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
10
20
30
br0 1 PVID Egress Untagged
'''harley$'''
But what is this? We have default VLAN <code>1 PVID Egress Untagged</code>. I don't want this. Seems setting <code>DefaultPVID=none</code> in 08-br0.netdev doesn't work. I've made a [[#Workaround for setting DefaultPVID=none|Workaround for setting DefaultPVID=none]]. Looking at this behavior I found that we can set <code>default_pvid</code> in the kernel only if <code>vlan_filtering = 0</code>. By hand I have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
If I start a guest I will get now:
'''harley$''' virsh start deb9-test
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 10
20
30
br0 None
vnet0 None
'''harley$'''
The virtual network interface vnet0 for deb9-test has no VLAN ID. Libvirt does not know something about this so we have to tell it. Libvirt provides [https://www.libvirt.org/hooks.html hook scripts] that we can use for this. We have to:
# [[#define VLAN-ID the virtual machine belongs to]]
# [[#get information on startup from the runtime XML-config of the domain]]
# [[#set VLAN-ID to the dynamic virtual network interface vnet*]]
For debugging the hook-scripts I've made a small script:
'''harley$''' cat /etc/libvirt/hooks/debug.sh
#!/bin/bash -e
# https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.
# For debug set symlink to hook-script daemon, qemu, lxc, libxl and/or network,
# e.g. 'sudo ln -s debug.sh qemu' and restart libvirtd.

logfile='/var/log/libvirt/hooks.log'

echo "$0" >>$logfile
date -Iseconds >>$logfile
echo "\$1=$1, \$2=$2, \$3=$3, \$4=$4" >>$logfile
cat - >>$logfile
echo -e "\n---------------------------------------------" >>$logfile
'''harley$'''

=== define VLAN-ID the virtual machine belongs to ===
For this we have an extra [https://libvirt.org/formatdomain.html#elementsMetadata element <metadata> in Domain XML format] for custom metadata. We can simply add the information to the static configuration with <code>'''harley$''' virsh edit deb9-test</code> like this (look only at the <metadata> element):
'''harley$''' virsh dumpxml deb9-test | head -n9
<domain type='kvm' id='1'>
<name>deb9-test</name>
<uuid>70d56a28-795d-4010-9403-513a4bd6b66a</uuid>
<metadata>
<my:home xmlns:my="http://hoeft-online.de/my/">
<my:vlan>10</my:vlan>
</my:home>
</metadata>
<memory unit='KiB'>1048576</memory>

=== get information on startup from the runtime XML-config of the domain ===
It seems a little bit difficult to get needed information out of the big XML-config but it's no problem with XSLT. I've made a XSL-stylesheet for this and use xmlstarlet. Start a virtual machine and then its runtime configuration is available with <code>'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr qemu.xsl</code>. With this I can test my stylesheet. Here is it:
'''harley$''' cat /etc/libvirt/hooks/qemu.xsl
<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:my="http://hoeft-online.de/my/" exclude-result-prefixes="my">
<xsl:output omit-xml-declaration="yes" indent="no"
encoding="utf-8" media-type="text/xml"/>
<xsl:strip-space elements="*"/>
<xsl:template match="text()|@*"/>

<xsl:template match="/domain">
<meta>
<xsl:apply-templates/>
</meta>
</xsl:template>

<xsl:template match="metadata/my:home/my:vlan">
<vlan>
<xsl:value-of select="."/>
</vlan>
</xsl:template>

<xsl:template match='interface[@type="bridge"]/target'>
<dev>
<xsl:value-of select="@dev"/>
</dev>
</xsl:template>

</xsl:stylesheet>
'''harley$'''

'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr /etc/libvirt/hooks/qemu.xsl
<meta><vlan>10</vlan><dev>vnet0</dev></meta>'''harley$'''

=== set VLAN-ID to the dynamic virtual network interface vnet* ===
Putting it all together here is the executable hook-script:
'''harley$''' cat /etc/libvirt/hooks/qemu
#!/bin/bash
#/etc/libvirt/hooks/qemu
# Docs: https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.

# On startup of the domain (guest) This script does:
# Get Metadata VLAN-ID of the guest and target device of the bridge from
# the domain-xml available on standard input. It is the runtime
# version from 'virsh dumpxml domainname'. For extracting the
# information we use a XSL-stylesheet. Example input into $META:
# <meta><vlan>10</vlan><dev>vnet0</dev></meta>
# Select $DEV from $META
# Select $VLAN from $META
# Set $VLAN to $DEV on the bridge

case "$2" in
prepare)
;;
start)
META=$(/usr/bin/xmlstarlet tr /etc/libvirt/hooks/qemu.xsl -)
DEV=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/dev')
VLAN=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/vlan')
if [[ -n $DEV && -n $VLAN ]]; then
/sbin/bridge vlan add vid "$VLAN" dev "$DEV"
fi
;;
started)
;;
stopped)
;;
release)
;;
migrate)
;;
restore)
;;
reconnect)
;;
attach)
;;
*)
echo "qemu hook called with unexpected options $*" >&2
exit 1
;;
esac
'''harley$''' sudo chmod 744 /etc/libvirt/hooks/qemu
'''harley$'''

=== References ===
* https://www.libvirt.org/hooks.html
* https://serverfault.com/questions/696011/libvirt-hook-qemu-suse12

== Workaround for setting DefaultPVID=none ==
Setting [https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#DefaultPVID= DefaultPVID] in a<code>systemd-networkd</code> configuration file to "none" does not work. Until this bug is fixed I've made a workaround. The kernel accepts setting <code>default_pvid</code> to 0 (means "none") only if <code>vlan_filtering=0</code>, so we have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
Check with listing of [[#bridge-settings|bridge-settings]].
Theese commands must run with <code>systemd-networkd</code> so we need a service for this. First I make a script and make it executable for root:
'''harley$''' cat /etc/systemd/network/DefaultPVID.sh
#!/bin/bash
#echo "entering DefaultPVID.sh" >>/tmp/debug.log

BRDIR="/sys/class/net/br0/bridge/"

if [[ -f $BRDIR/vlan_filtering && -f $BRDIR/default_pvid ]]; then
#echo "setting DefaultPVID" >>/tmp/debug.log
VLAN_FILTERING="$(cat "$BRDIR"/vlan_filtering)"
echo 0 >"$BRDIR"/vlan_filtering
echo 0 >"$BRDIR"/default_pvid
echo "$VLAN_FILTERING" >"$BRDIR"/vlan_filtering
fi
exit 0
'''harley$''' sudo chmod 744 /etc/systemd/network/DefaultPVID.sh
'''harley$'''
Next I create a service to execute this script:
'''harley$''' cat /etc/systemd/system/DefaultPVID.service
[Unit]
Description=set DefaultPVID on a bridge as workaround
Wants=network.target
After=network.target

[Service]
Type=oneshot
ExecStart=/etc/systemd/network/DefaultPVID.sh

[Install]
WantedBy=multi-user.target
'''harley$'''
And then I create a [https://www.freedesktop.org/software/systemd/man/systemd.unit.html#id-1.11.3 drop-in file for overriding vendor settings] so this service will be executed together with <code>systemd-networkd</code>:
'''harley$''' cat /etc/systemd/system/systemd-networkd.service.d/DefaultPVID.conf
[Unit]
# This is only a workaround. DefaultPVID cannot be set in
# /etc/systemd/network/br0.netdev. It seems buggy.
Wants=DefaultPVID.service
Before=DefaultPVID.service
'''harley$'''

[[Category:Virtualization]]

VLAN for virtual machines

2017-09-27T21:31:18Z

Ingo: linkfix

== Introduction ==
I wanted to update VLAN connections for virtual
machines to newer technologies and put a question on
[https://unix.stackexchange.com/questions/392758/setup-vlan-on-linux-bridge-for-virtual-machines-with-systemd unix.stackexchange]. But I do not get any answer. It seems there is very
little knowledge for this out there. So I decided to work on it by myself
and document it here.

In gerneral I will look at three methods:
# [[#oldstyle linux bridge as hub|oldstyle linux bridge as hub]]
# [[#linux bridge as hub|linux bridge as hub]]
# [[#linux bridge with libvirt hook scripts|linux bridge with libvirt hook scripts]]

== Preparation ==
I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: [[Setup KVM with console]]. I'm sitting on harley as host, my all day workstation.
Now I start the virtual machine, login and show its interface setting:
'''harley$''' virsh start --console deb9-test
login
'''deb9-test$''' cat /etc/systemd/network/08-vlan10.netdev
[NetDev]
Name=vlan10
Kind=vlan
[VLAN]
Id=10
'''deb9-test$''' cat /etc/systemd/network/12-vlan10_attach-to-if.network
[Match]
Name=ens2
[Network]
VLAN=vlan10
'''deb9-test$''' cat /etc/systemd/network/16-vlan10_up.network
[Match]
Name=vlan10
[Network]
DHCP=ipv4
IPv6AcceptRA=no
LinkLocalAddressing=no
To test if the virtual machine has connection I use:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. --
Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed
Sep 23 20:34:05 systemd[1]: Started Network Service.
Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready
Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success
Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier
Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier
Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL
Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL
Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1
Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured
Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured
'''deb9-test$'''
4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. --
Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed
Sep 23 20:44:59 systemd[1]: Started Network Service.
Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready
Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success
Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier
Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier
Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL
Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL
Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured
'''deb9-test$'''

Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest.
'''deb9-test$''' grep ^ExecStart= /lib/systemd/system/serial-getty@.service
ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM
modify to
ExecStart=-/sbin/agetty --autologin ''yourloginname'' --keep-baud 115200,38400,9600 %I $TERM
To list all settings of the bridge you can use:
'''harley$''' find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort

== oldstyle linux bridge as hub ==
This works always with the old linux bridge that do not know anything about VLAN. The trick is to set it to a complete transparent state for all connected interfaces like a hub. But you have to know that the bridge will then forward all packets to all interfaces simultanously. You can do it by setting the ageing time to 0.

Disable systemd-networkd and start networking with ifupdown:
'''harley$''' sudo systemctl stop systemd-networkd
Warning: Stopping systemd-networkd.service, but it can still be activated by:
systemd-networkd.socket
'''harley$''' sudo systemctl disable systemd-networkd
Removed /etc/systemd/system/multi-user.target.wants/systemd-networkd.service.
Removed /etc/systemd/system/sockets.target.wants/systemd-networkd.socket.
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable networking
'''harley$''' sudo systemctl start networking.service
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/network/interfaces
auto br0
iface br0 inet manual
bridge_ports enp1s0
bridge_ageing 0
bridge_stp off
'''harley$''' sudo ifup br0
Waiting for br0 to get ready (MAXWAIT is 32 seconds).
'''harley$'''
It's all in place now:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets.

=== References ===
* https://wiki.debian.org/NetworkConfiguration#Bridging_without_Switching

== linux bridge as hub ==
Now I try to setup [[#oldstyle linux bridge as hub]] just with systemd-networkd.

Disable networking with ifupdown and start systemd-networkd:
'''harley$''' sudo systemctl stop networking.service
'''harley$''' sudo systemctl disable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable networking
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable systemd-networkd
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /lib/systemd/system/systemd-networkd.socket.
'''harley$''' sudo systemctl start systemd-networkd
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
AgeingTimeSec=0
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl restart systemd-networkd
'''harley$'''

AgeingTimeSec=0 is not acepted but should:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
30000 (means 300 sec)
'''harley$'''
But I've found a workaround. Useing a number between '''.'''01 and '''.'''000001 (there are dots) will set ageing_time to 0.
So set AgeingTimeSec='''.'''000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we
will get:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
'''harley$'''
The guest gets now an IP-Address on boot and is connected to VLAN 10.

=== Discussion ===
This works because of [[#References|three conditions]].
# ageing time is 0: ageing time specifies the number of seconds a MAC Address will be kept in the forwarding database after having a packet received from this MAC Address. Setting it to 0 means there is never a MAC Address stored in the FDB.
# unicast flood on interfaces is on: this controls whether the bridge should flood traffic for which an FDB entry is missing and the destination is unknown through this port. Defaults to on.
# spanning tree protocol (stp) is disabled: we don't have a forward_delay at startup for the learning phase of spanning tree.
I have a running and connected virtual machine:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
br0 1 PVID Egress Untagged
vnet0 1 PVID Egress Untagged
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/forward_delay
1500
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
Indeed we have forward_delay 1500 (means 15 sec) but it doesn't matter. stp_state is 0 (disabled), no spanning tree. Flood (means unicast flood) is on as I can see:
'''harley$''' sudo bridge -d link show
''3: enp1s0'' state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
''95: vnet0'' state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
'''harley$'''

Let's have a look at flooding on the interfaces. I disable it on the physical interface enp1s0 of the bridge and reboot the guest:
'''harley$''' sudo bridge link set dev enp1s0 flood off
'''harley$'''
The guest gets an IP-Address from the DHCP-Server but then can't ping its gateway. DHCP-REQUEST is broadcast and goes thru enp1s0. DHCP-ANSWER comes back thru it to any other (here only vnet0) interface which has flood on. Ping is unicast and isn't forwareded on enp1s0. If I set enp1s0 flood on and vnet0 flood off and <code>'''deb9-test$ '''sudo systemctl restart systemd-networkd</code>, I get no IP-Address from DHCP-Server and can't ping the interface. Incoming DHCP-ANSWER isn't broadcast and vnet0 doesn't forward it to the guest.

Btw. this method has bad performance as we can see with monitor. We insert MAC-Addresses into FDB for just deleting it immediately, all for nothing.
'''harley$''' sudo bridge monitor fdb
52:54:00:01:76:20 dev enp1s0 master br0
52:54:00:b0:ca:63 dev vnet0 master br0
f4:f2:6d:2c:87:f7 dev enp1s0 master br0
00:80:3f:2a:31:1a dev enp1s0 master br0
Deleted 52:54:00:01:76:20 dev enp1s0 master br0 stale
Deleted 52:54:00:b0:ca:63 dev vnet0 master br0 stale
Deleted 00:80:3f:2a:31:1a dev enp1s0 master br0 stale
Deleted f4:f2:6d:2c:87:f7 dev enp1s0 master br0 stale
...

=== References ===
* https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
* https://www.freedesktop.org/software/systemd/man/systemd.network.html

== linux bridge with libvirt hook scripts ==
We setup a bridge with VLAN enabled:
'''harley$''' cat 08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
DefaultPVID=none
VLANFiltering=true
STP=false
'''harley$''' cat 12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
[BridgeVLAN]
VLAN=10
[BridgeVLAN]
VLAN=20
[BridgeVLAN]
VLAN=30
'''harley$''' cat 16-br0_up.network
[Match]
Name=br0
With this I get:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
10
20
30
br0 1 PVID Egress Untagged
'''harley$'''
But what is this? We have default VLAN <code>1 PVID Egress Untagged</code>. I don't want this. Seems setting <code>DefaultPVID=none</code> in 08-br0.netdev doesn't work. I've made a [[#Workaround for setting DefaultPVID=none|Workaround for setting DefaultPVID=none]]. Looking at this behavior I found that we can set <code>default_pvid</code> in the kernel only if <code>vlan_filtering = 0</code>. By hand I have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
If I start a guest I will get now:
'''harley$''' virsh start deb9-test
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 10
20
30
br0 None
vnet0 None
'''harley$'''
The virtual network interface vnet0 for deb9-test has no VLAN ID. Libvirt does not know something about this so we have to tell it. Libvirt provides [https://www.libvirt.org/hooks.html hook scripts] that we can use for this. We have to:
# [[#define VLAN-ID the virtual machine belongs to]]
# [[#get information on startup from the runtime XML-config of the domain]]
# [[#set VLAN-ID to the dynamic virtual network interface vnet*]]
For debugging the hook-scripts I've made a small script:
'''harley$''' cat debug.sh
#!/bin/bash -e
# https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.
# For debug set symlink to hook-script daemon, qemu, lxc, libxl and/or network,
# e.g. 'sudo ln -s debug.sh qemu' and restart libvirtd.

logfile='/var/log/libvirt/hooks.log'

echo "$0" >>$logfile
date -Iseconds >>$logfile
echo "\$1=$1, \$2=$2, \$3=$3, \$4=$4" >>$logfile
cat - >>$logfile
echo -e "\n---------------------------------------------" >>$logfile
'''harley$'''

=== define VLAN-ID the virtual machine belongs to ===
For this we have an extra [https://libvirt.org/formatdomain.html#elementsMetadata element <metadata> in Domain XML format] for custom metadata. We can simply add the information to the static configuration with <code>'''harley$''' virsh edit deb9-test</code> like this (look only at the <metadata> element):
'''harley$''' virsh dumpxml deb9-test | head -n9
<domain type='kvm' id='1'>
<name>deb9-test</name>
<uuid>70d56a28-795d-4010-9403-513a4bd6b66a</uuid>
<metadata>
<my:home xmlns:my="http://hoeft-online.de/my/">
<my:vlan>10</my:vlan>
</my:home>
</metadata>
<memory unit='KiB'>1048576</memory>

=== get information on startup from the runtime XML-config of the domain ===
It seems a little bit difficult to get needed information out of the big XML-config but it's no problem with XSLT. I've made a XSL-stylesheet for this and use xmlstarlet. Start a virtual machine and then its runtime configuration is available with <code>'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr qemu.xsl</code>. With this I can test my stylesheet. Here is it:
'''harley$''' cat qemu.xsl
<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:my="http://hoeft-online.de/my/" exclude-result-prefixes="my">
<xsl:output omit-xml-declaration="yes" indent="no"
encoding="utf-8" media-type="text/xml"/>
<xsl:strip-space elements="*"/>
<xsl:template match="text()|@*"/>

<xsl:template match="/domain">
<meta>
<xsl:apply-templates/>
</meta>
</xsl:template>

<xsl:template match="metadata/my:home/my:vlan">
<vlan>
<xsl:value-of select="."/>
</vlan>
</xsl:template>

<xsl:template match='interface[@type="bridge"]/target'>
<dev>
<xsl:value-of select="@dev"/>
</dev>
</xsl:template>

</xsl:stylesheet>
'''harley$'''

'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr /etc/libvirt/hooks/qemu.xsl
<meta><vlan>10</vlan><dev>vnet0</dev></meta>'''harley$'''

=== set VLAN-ID to the dynamic virtual network interface vnet* ===
Putting it all together here is the executable hook-script:
'''harley$''' cat /etc/libvirt/hooks/qemu
#!/bin/bash
#/etc/libvirt/hooks/qemu
# Docs: https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.

# On startup of the domain (guest) This script does:
# Get Metadata VLAN-ID of the guest and target device of the bridge from
# the domain-xml available on standard input. It is the runtime
# version from 'virsh dumpxml domainname'. For extracting the
# information we use a XSL-stylesheet. Example input into $META:
# <meta><vlan>10</vlan><dev>vnet0</dev></meta>
# Select $DEV from $META
# Select $VLAN from $META
# Set $VLAN to $DEV on the bridge

case "$2" in
prepare)
;;
start)
META=$(/usr/bin/xmlstarlet tr /etc/libvirt/hooks/qemu.xsl -)
DEV=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/dev')
VLAN=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/vlan')
if [[ -n $DEV && -n $VLAN ]]; then
/sbin/bridge vlan add vid "$VLAN" dev "$DEV"
fi
;;
started)
;;
stopped)
;;
release)
;;
migrate)
;;
restore)
;;
reconnect)
;;
attach)
;;
*)
echo "qemu hook called with unexpected options $*" >&2
exit 1
;;
esac
'''harley$''' sudo chmod 744 /etc/libvirt/hooks/qemu
'''harley$'''

=== References ===
* https://www.libvirt.org/hooks.html
* https://serverfault.com/questions/696011/libvirt-hook-qemu-suse12

== Workaround for setting DefaultPVID=none ==
Setting [https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#DefaultPVID= DefaultPVID] in a<code>systemd-networkd</code> configuration file to "none" does not work. Until this bug is fixed I've made a workaround. The kernel accepts setting <code>default_pvid</code> to 0 (means "none") only if <code>vlan_filtering=0</code>, so we have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
Check with listing of [[#bridge-settings|bridge-settings]].
Theese commands must run with <code>systemd-networkd</code> so we need a service for this. First I make a script and make it executable for root:
'''harley$''' cat /etc/systemd/network/DefaultPVID.sh
#!/bin/bash
#echo "entering DefaultPVID.sh" >>/tmp/debug.log

BRDIR="/sys/class/net/br0/bridge/"

if [[ -f $BRDIR/vlan_filtering && -f $BRDIR/default_pvid ]]; then
#echo "setting DefaultPVID" >>/tmp/debug.log
VLAN_FILTERING="$(cat "$BRDIR"/vlan_filtering)"
echo 0 >"$BRDIR"/vlan_filtering
echo 0 >"$BRDIR"/default_pvid
echo "$VLAN_FILTERING" >"$BRDIR"/vlan_filtering
fi
exit 0
'''harley$''' sudo chmod 744 /etc/systemd/network/DefaultPVID.sh
'''harley$'''
Next I create a service to execute this script:
'''harley$''' cat /etc/systemd/system/DefaultPVID.service
[Unit]
Description=set DefaultPVID on a bridge as workaround
Wants=network.target
After=network.target

[Service]
Type=oneshot
ExecStart=/etc/systemd/network/DefaultPVID.sh

[Install]
WantedBy=multi-user.target
'''harley$'''
And then I create a [https://www.freedesktop.org/software/systemd/man/systemd.unit.html#id-1.11.3 drop-in file for overriding vendor settings] so this service will be executed together with <code>systemd-networkd</code>:
'''harley$''' cat /etc/systemd/system/systemd-networkd.service.d/DefaultPVID.conf
[Unit]
# This is only a workaround. DefaultPVID cannot be set in
# /etc/systemd/network/br0.netdev. It seems buggy.
Wants=DefaultPVID.service
Before=DefaultPVID.service
'''harley$'''

[[Category:Virtualization]]

VLAN for virtual machines

2017-09-27T21:28:31Z

Ingo: describe workaround setting DefaultPVID

== Introduction ==
I wanted to update VLAN connections for virtual
machines to newer technologies and put a question on
[https://unix.stackexchange.com/questions/392758/setup-vlan-on-linux-bridge-for-virtual-machines-with-systemd unix.stackexchange]. But I do not get any answer. It seems there is very
little knowledge for this out there. So I decided to work on it by myself
and document it here.

In gerneral I will look at four methods:
# [[#oldstyle linux bridge as hub|oldstyle linux bridge as hub]]
# [[#linux bridge as hub|linux bridge as hub]]
# [[#linux bridge with libvirt hook scripts|linux bridge with libvirt hook scripts]]
# [[#Open vSwitch|Open vSwitch]]

== Preparation ==
I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: [[Setup KVM with console]]. I'm sitting on harley as host, my all day workstation.
Now I start the virtual machine, login and show its interface setting:
'''harley$''' virsh start --console deb9-test
login
'''deb9-test$''' cat /etc/systemd/network/08-vlan10.netdev
[NetDev]
Name=vlan10
Kind=vlan
[VLAN]
Id=10
'''deb9-test$''' cat /etc/systemd/network/12-vlan10_attach-to-if.network
[Match]
Name=ens2
[Network]
VLAN=vlan10
'''deb9-test$''' cat /etc/systemd/network/16-vlan10_up.network
[Match]
Name=vlan10
[Network]
DHCP=ipv4
IPv6AcceptRA=no
LinkLocalAddressing=no
To test if the virtual machine has connection I use:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. --
Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed
Sep 23 20:34:05 systemd[1]: Started Network Service.
Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready
Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success
Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier
Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier
Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL
Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL
Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1
Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured
Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured
'''deb9-test$'''
4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. --
Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed
Sep 23 20:44:59 systemd[1]: Started Network Service.
Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready
Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success
Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier
Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier
Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL
Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL
Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured
'''deb9-test$'''

Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest.
'''deb9-test$''' grep ^ExecStart= /lib/systemd/system/serial-getty@.service
ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM
modify to
ExecStart=-/sbin/agetty --autologin ''yourloginname'' --keep-baud 115200,38400,9600 %I $TERM
To list all settings of the bridge you can use:
'''harley$''' find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort

== oldstyle linux bridge as hub ==
This works always with the old linux bridge that do not know anything about VLAN. The trick is to set it to a complete transparent state for all connected interfaces like a hub. But you have to know that the bridge will then forward all packets to all interfaces simultanously. You can do it by setting the ageing time to 0.

Disable systemd-networkd and start networking with ifupdown:
'''harley$''' sudo systemctl stop systemd-networkd
Warning: Stopping systemd-networkd.service, but it can still be activated by:
systemd-networkd.socket
'''harley$''' sudo systemctl disable systemd-networkd
Removed /etc/systemd/system/multi-user.target.wants/systemd-networkd.service.
Removed /etc/systemd/system/sockets.target.wants/systemd-networkd.socket.
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable networking
'''harley$''' sudo systemctl start networking.service
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/network/interfaces
auto br0
iface br0 inet manual
bridge_ports enp1s0
bridge_ageing 0
bridge_stp off
'''harley$''' sudo ifup br0
Waiting for br0 to get ready (MAXWAIT is 32 seconds).
'''harley$'''
It's all in place now:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets.

=== References ===
* https://wiki.debian.org/NetworkConfiguration#Bridging_without_Switching

== linux bridge as hub ==
Now I try to setup [[#oldstyle linux bridge as hub]] just with systemd-networkd.

Disable networking with ifupdown and start systemd-networkd:
'''harley$''' sudo systemctl stop networking.service
'''harley$''' sudo systemctl disable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable networking
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable systemd-networkd
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /lib/systemd/system/systemd-networkd.socket.
'''harley$''' sudo systemctl start systemd-networkd
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
AgeingTimeSec=0
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl restart systemd-networkd
'''harley$'''

AgeingTimeSec=0 is not acepted but should:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
30000 (means 300 sec)
'''harley$'''
But I've found a workaround. Useing a number between '''.'''01 and '''.'''000001 (there are dots) will set ageing_time to 0.
So set AgeingTimeSec='''.'''000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we
will get:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
'''harley$'''
The guest gets now an IP-Address on boot and is connected to VLAN 10.

=== Discussion ===
This works because of [[#References|three conditions]].
# ageing time is 0: ageing time specifies the number of seconds a MAC Address will be kept in the forwarding database after having a packet received from this MAC Address. Setting it to 0 means there is never a MAC Address stored in the FDB.
# unicast flood on interfaces is on: this controls whether the bridge should flood traffic for which an FDB entry is missing and the destination is unknown through this port. Defaults to on.
# spanning tree protocol (stp) is disabled: we don't have a forward_delay at startup for the learning phase of spanning tree.
I have a running and connected virtual machine:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
br0 1 PVID Egress Untagged
vnet0 1 PVID Egress Untagged
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/forward_delay
1500
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
Indeed we have forward_delay 1500 (means 15 sec) but it doesn't matter. stp_state is 0 (disabled), no spanning tree. Flood (means unicast flood) is on as I can see:
'''harley$''' sudo bridge -d link show
''3: enp1s0'' state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
''95: vnet0'' state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
'''harley$'''

Let's have a look at flooding on the interfaces. I disable it on the physical interface enp1s0 of the bridge and reboot the guest:
'''harley$''' sudo bridge link set dev enp1s0 flood off
'''harley$'''
The guest gets an IP-Address from the DHCP-Server but then can't ping its gateway. DHCP-REQUEST is broadcast and goes thru enp1s0. DHCP-ANSWER comes back thru it to any other (here only vnet0) interface which has flood on. Ping is unicast and isn't forwareded on enp1s0. If I set enp1s0 flood on and vnet0 flood off and <code>'''deb9-test$ '''sudo systemctl restart systemd-networkd</code>, I get no IP-Address from DHCP-Server and can't ping the interface. Incoming DHCP-ANSWER isn't broadcast and vnet0 doesn't forward it to the guest.

Btw. this method has bad performance as we can see with monitor. We insert MAC-Addresses into FDB for just deleting it immediately, all for nothing.
'''harley$''' sudo bridge monitor fdb
52:54:00:01:76:20 dev enp1s0 master br0
52:54:00:b0:ca:63 dev vnet0 master br0
f4:f2:6d:2c:87:f7 dev enp1s0 master br0
00:80:3f:2a:31:1a dev enp1s0 master br0
Deleted 52:54:00:01:76:20 dev enp1s0 master br0 stale
Deleted 52:54:00:b0:ca:63 dev vnet0 master br0 stale
Deleted 00:80:3f:2a:31:1a dev enp1s0 master br0 stale
Deleted f4:f2:6d:2c:87:f7 dev enp1s0 master br0 stale
...

=== References ===
* https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
* https://www.freedesktop.org/software/systemd/man/systemd.network.html

== linux bridge with libvirt hook scripts ==
We setup a bridge with VLAN enabled:
'''harley$''' cat 08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
DefaultPVID=none
VLANFiltering=true
STP=false
'''harley$''' cat 12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
[BridgeVLAN]
VLAN=10
[BridgeVLAN]
VLAN=20
[BridgeVLAN]
VLAN=30
'''harley$''' cat 16-br0_up.network
[Match]
Name=br0
With this I get:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
10
20
30
br0 1 PVID Egress Untagged
'''harley$'''
But what is this? We have default VLAN <code>1 PVID Egress Untagged</code>. I don't want this. Seems setting <code>DefaultPVID=none</code> in 08-br0.netdev doesn't work. I've made a [[#Workaround for setting DefaultPVID=none|Workaround for setting DefaultPVID=none]]. Looking at this behavior I found that we can set <code>default_pvid</code> in the kernel only if <code>vlan_filtering = 0</code>. By hand I have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
If I start a guest I will get now:
'''harley$''' virsh start deb9-test
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 10
20
30
br0 None
vnet0 None
'''harley$'''
The virtual network interface vnet0 for deb9-test has no VLAN ID. Libvirt does not know something about this so we have to tell it. Libvirt provides [https://www.libvirt.org/hooks.html hook scripts] that we can use for this. We have to:
# [[#define VLAN-ID the virtual machine belongs to]]
# [[#get information on startup from the runtime XML-config of the domain]]
# [[#set VLAN-ID to the dynamic virtual network interface vnet*]]
For debugging the hook-scripts I've made a small script:
'''harley$''' cat debug.sh
#!/bin/bash -e
# https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.
# For debug set symlink to hook-script daemon, qemu, lxc, libxl and/or network,
# e.g. 'sudo ln -s debug.sh qemu' and restart libvirtd.

logfile='/var/log/libvirt/hooks.log'

echo "$0" >>$logfile
date -Iseconds >>$logfile
echo "\$1=$1, \$2=$2, \$3=$3, \$4=$4" >>$logfile
cat - >>$logfile
echo -e "\n---------------------------------------------" >>$logfile
'''harley$'''

=== define VLAN-ID the virtual machine belongs to ===
For this we have an extra [https://libvirt.org/formatdomain.html#elementsMetadata element <metadata> in Domain XML format] for custom metadata. We can simply add the information to the static configuration with <code>'''harley$''' virsh edit deb9-test</code> like this (look only at the <metadata> element):
'''harley$''' virsh dumpxml deb9-test | head -n9
<domain type='kvm' id='1'>
<name>deb9-test</name>
<uuid>70d56a28-795d-4010-9403-513a4bd6b66a</uuid>
<metadata>
<my:home xmlns:my="http://hoeft-online.de/my/">
<my:vlan>10</my:vlan>
</my:home>
</metadata>
<memory unit='KiB'>1048576</memory>

=== get information on startup from the runtime XML-config of the domain ===
It seems a little bit difficult to get needed information out of the big XML-config but it's no problem with XSLT. I've made a XSL-stylesheet for this and use xmlstarlet. Start a virtual machine and then its runtime configuration is available with <code>'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr qemu.xsl</code>. With this I can test my stylesheet. Here is it:
'''harley$''' cat qemu.xsl
<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:my="http://hoeft-online.de/my/" exclude-result-prefixes="my">
<xsl:output omit-xml-declaration="yes" indent="no"
encoding="utf-8" media-type="text/xml"/>
<xsl:strip-space elements="*"/>
<xsl:template match="text()|@*"/>

<xsl:template match="/domain">
<meta>
<xsl:apply-templates/>
</meta>
</xsl:template>

<xsl:template match="metadata/my:home/my:vlan">
<vlan>
<xsl:value-of select="."/>
</vlan>
</xsl:template>

<xsl:template match='interface[@type="bridge"]/target'>
<dev>
<xsl:value-of select="@dev"/>
</dev>
</xsl:template>

</xsl:stylesheet>
'''harley$'''

'''harley$''' virsh dumpxml deb9-test | xmlstarlet tr /etc/libvirt/hooks/qemu.xsl
<meta><vlan>10</vlan><dev>vnet0</dev></meta>'''harley$'''

=== set VLAN-ID to the dynamic virtual network interface vnet* ===
Putting it all together here is the executable hook-script:
'''harley$''' cat /etc/libvirt/hooks/qemu
#!/bin/bash
#/etc/libvirt/hooks/qemu
# Docs: https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.

# On startup of the domain (guest) This script does:
# Get Metadata VLAN-ID of the guest and target device of the bridge from
# the domain-xml available on standard input. It is the runtime
# version from 'virsh dumpxml domainname'. For extracting the
# information we use a XSL-stylesheet. Example input into $META:
# <meta><vlan>10</vlan><dev>vnet0</dev></meta>
# Select $DEV from $META
# Select $VLAN from $META
# Set $VLAN to $DEV on the bridge

case "$2" in
prepare)
;;
start)
META=$(/usr/bin/xmlstarlet tr /etc/libvirt/hooks/qemu.xsl -)
DEV=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/dev')
VLAN=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/vlan')
if [[ -n $DEV && -n $VLAN ]]; then
/sbin/bridge vlan add vid "$VLAN" dev "$DEV"
fi
;;
started)
;;
stopped)
;;
release)
;;
migrate)
;;
restore)
;;
reconnect)
;;
attach)
;;
*)
echo "qemu hook called with unexpected options $*" >&2
exit 1
;;
esac
'''harley$''' sudo chmod 744 /etc/libvirt/hooks/qemu
'''harley$'''

=== References ===
* https://www.libvirt.org/hooks.html
* https://serverfault.com/questions/696011/libvirt-hook-qemu-suse12

== Workaround for setting DefaultPVID=none ==
Setting [https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#DefaultPVID= DefaultPVID] in a<code>systemd-networkd</code> configuration file to "none" does not work. Until this bug is fixed I've made a workaround. The kernel accepts setting <code>default_pvid</code> to 0 (means "none") only if <code>vlan_filtering=0</code>, so we have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
Check with listing of [[#bridge-settings|bridge-settings]].
Theese commands must run with <code>systemd-networkd</code> so we need a service for this. First I make a script and make it executable for root:
'''harley$''' cat /etc/systemd/network/DefaultPVID.sh
#!/bin/bash
#echo "entering DefaultPVID.sh" >>/tmp/debug.log

BRDIR="/sys/class/net/br0/bridge/"

if [[ -f $BRDIR/vlan_filtering && -f $BRDIR/default_pvid ]]; then
#echo "setting DefaultPVID" >>/tmp/debug.log
VLAN_FILTERING="$(cat "$BRDIR"/vlan_filtering)"
echo 0 >"$BRDIR"/vlan_filtering
echo 0 >"$BRDIR"/default_pvid
echo "$VLAN_FILTERING" >"$BRDIR"/vlan_filtering
fi
exit 0
'''harley$''' sudo chmod 744 /etc/systemd/network/DefaultPVID.sh
'''harley$'''
Next I create a service to execute this script:
'''harley$''' cat /etc/systemd/system/DefaultPVID.service
[Unit]
Description=set DefaultPVID on a bridge as workaround
Wants=network.target
After=network.target

[Service]
Type=oneshot
ExecStart=/etc/systemd/network/DefaultPVID.sh

[Install]
WantedBy=multi-user.target
'''harley$'''
And then I create a [https://www.freedesktop.org/software/systemd/man/systemd.unit.html#id-1.11.3 drop-in file for overriding vendor settings] so this service will be executed together with <code>systemd-networkd</code>:
'''harley$''' cat /etc/systemd/system/systemd-networkd.service.d/DefaultPVID.conf
[Unit]
# This is only a workaround. DefaultPVID cannot be set in
# /etc/systemd/network/br0.netdev. It seems buggy.
Wants=DefaultPVID.service
Before=DefaultPVID.service
'''harley$'''

[[Category:Virtualization]]

VLAN for virtual machines

2017-09-27T14:13:32Z

Ingo: describe bridge with hook scripts

== Introduction ==
I wanted to update VLAN connections for virtual
machines to newer technologies and put a question on
[https://unix.stackexchange.com/questions/392758/setup-vlan-on-linux-bridge-for-virtual-machines-with-systemd unix.stackexchange]. But I do not get any answer. It seems there is very
little knowledge for this out there. So I decided to work on it by myself
and document it here.

In gerneral I will look at four methods:
# [[#oldstyle linux bridge as hub|oldstyle linux bridge as hub]]
# [[#linux bridge as hub|linux bridge as hub]]
# [[#linux bridge with libvirt hook scripts|linux bridge with libvirt hook scripts]]
# [[#Open vSwitch|Open vSwitch]]

== Preparation ==
I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: [[Setup KVM with console]]. I'm sitting on harley as host, my all day workstation.
Now I start the virtual machine, login and show its interface setting:
'''harley$''' virsh start --console deb9-test
login
'''deb9-test$''' cat /etc/systemd/network/08-vlan10.netdev
[NetDev]
Name=vlan10
Kind=vlan
[VLAN]
Id=10
'''deb9-test$''' cat /etc/systemd/network/12-vlan10_attach-to-if.network
[Match]
Name=ens2
[Network]
VLAN=vlan10
'''deb9-test$''' cat /etc/systemd/network/16-vlan10_up.network
[Match]
Name=vlan10
[Network]
DHCP=ipv4
IPv6AcceptRA=no
LinkLocalAddressing=no
To test if the virtual machine has connection I use:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. --
Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed
Sep 23 20:34:05 systemd[1]: Started Network Service.
Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready
Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success
Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier
Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier
Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL
Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL
Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1
Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured
Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured
'''deb9-test$'''
4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this:
'''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service
-- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. --
Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed
Sep 23 20:44:59 systemd[1]: Started Network Service.
Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready
Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success
Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier
Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier
Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL
Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL
Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured
'''deb9-test$'''

Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest.
'''deb9-test$''' grep ^ExecStart= /lib/systemd/system/serial-getty@.service
ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM
modify to
ExecStart=-/sbin/agetty --autologin ''yourloginname'' --keep-baud 115200,38400,9600 %I $TERM
To list all settings of the bridge you can use:
'''harley$''' find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort

== oldstyle linux bridge as hub ==
This works always with the old linux bridge that do not know anything about VLAN. The trick is to set it to a complete transparent state for all connected interfaces like a hub. But you have to know that the bridge will then forward all packets to all interfaces simultanously. You can do it by setting the ageing time to 0.

Disable systemd-networkd and start networking with ifupdown:
'''harley$''' sudo systemctl stop systemd-networkd
Warning: Stopping systemd-networkd.service, but it can still be activated by:
systemd-networkd.socket
'''harley$''' sudo systemctl disable systemd-networkd
Removed /etc/systemd/system/multi-user.target.wants/systemd-networkd.service.
Removed /etc/systemd/system/sockets.target.wants/systemd-networkd.socket.
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable networking
'''harley$''' sudo systemctl start networking.service
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/network/interfaces
auto br0
iface br0 inet manual
bridge_ports enp1s0
bridge_ageing 0
bridge_stp off
'''harley$''' sudo ifup br0
Waiting for br0 to get ready (MAXWAIT is 32 seconds).
'''harley$'''
It's all in place now:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets.

=== References ===
* https://wiki.debian.org/NetworkConfiguration#Bridging_without_Switching

== linux bridge as hub ==
Now I try to setup [[#oldstyle linux bridge as hub]] just with systemd-networkd.

Disable networking with ifupdown and start systemd-networkd:
'''harley$''' sudo systemctl stop networking.service
'''harley$''' sudo systemctl disable networking.service
Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable networking
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl enable systemd-networkd
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /lib/systemd/system/systemd-networkd.service.
Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /lib/systemd/system/systemd-networkd.socket.
'''harley$''' sudo systemctl start systemd-networkd
'''harley$'''
Setup the bridge and start it:
'''harley$''' cat /etc/systemd/network/08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
AgeingTimeSec=0
STP=false
'''harley$''' cat /etc/systemd/network/12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
'''harley$''' cat /etc/systemd/network/16-br0_up.network
[Match]
Name=br0
'''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0
'''harley$''' sudo systemctl restart systemd-networkd
'''harley$'''

AgeingTimeSec=0 is not acepted but should:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
30000 (means 300 sec)
'''harley$'''
But I've found a workaround. Useing a number between '''.'''01 and '''.'''000001 (there are dots) will set ageing_time to 0.
So set AgeingTimeSec='''.'''000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we
will get:
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
'''harley$''' cat /sys/class/net/br0/bridge/vlan_filtering
0
'''harley$'''
The guest gets now an IP-Address on boot and is connected to VLAN 10.

=== Discussion ===
This works because of [[#References|three conditions]].
# ageing time is 0: ageing time specifies the number of seconds a MAC Address will be kept in the forwarding database after having a packet received from this MAC Address. Setting it to 0 means there is never a MAC Address stored in the FDB.
# unicast flood on interfaces is on: this controls whether the bridge should flood traffic for which an FDB entry is missing and the destination is unknown through this port. Defaults to on.
# spanning tree protocol (stp) is disabled: we don't have a forward_delay at startup for the learning phase of spanning tree.
I have a running and connected virtual machine:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
br0 1 PVID Egress Untagged
vnet0 1 PVID Egress Untagged
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time
0
'''harley$''' cat /sys/class/net/br0/bridge/forward_delay
1500
'''harley$''' cat /sys/class/net/br0/bridge/stp_state
0
Indeed we have forward_delay 1500 (means 15 sec) but it doesn't matter. stp_state is 0 (disabled), no spanning tree. Flood (means unicast flood) is on as I can see:
'''harley$''' sudo bridge -d link show
''3: enp1s0'' state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
''95: vnet0'' state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on
'''harley$'''

Let's have a look at flooding on the interfaces. I disable it on the physical interface enp1s0 of the bridge and reboot the guest:
'''harley$''' sudo bridge link set dev enp1s0 flood off
'''harley$'''
The guest gets an IP-Address from the DHCP-Server but then can't ping its gateway. DHCP-REQUEST is broadcast and goes thru enp1s0. DHCP-ANSWER comes back thru it to any other (here only vnet0) interface which has flood on. Ping is unicast and isn't forwareded on enp1s0. If I set enp1s0 flood on and vnet0 flood off and <code>'''deb9-test$ '''sudo systemctl restart systemd-networkd</code>, I get no IP-Address from DHCP-Server and can't ping the interface. Incoming DHCP-ANSWER isn't broadcast and vnet0 doesn't forward it to the guest.

Btw. this method has bad performance as we can see with monitor. We insert MAC-Addresses into FDB for just deleting it immediately, all for nothing.
'''harley$''' sudo bridge monitor fdb
52:54:00:01:76:20 dev enp1s0 master br0
52:54:00:b0:ca:63 dev vnet0 master br0
f4:f2:6d:2c:87:f7 dev enp1s0 master br0
00:80:3f:2a:31:1a dev enp1s0 master br0
Deleted 52:54:00:01:76:20 dev enp1s0 master br0 stale
Deleted 52:54:00:b0:ca:63 dev vnet0 master br0 stale
Deleted 00:80:3f:2a:31:1a dev enp1s0 master br0 stale
Deleted f4:f2:6d:2c:87:f7 dev enp1s0 master br0 stale
...

=== References ===
* https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
* https://www.freedesktop.org/software/systemd/man/systemd.network.html

== linux bridge with libvirt hook scripts ==
We setup a bridge with VLAN enabled:
'''harley$''' cat 08-br0.netdev
[NetDev]
Name=br0
Kind=bridge
[Bridge]
DefaultPVID=none
VLANFiltering=true
STP=false
'''harley$''' cat 12-br0_add-enp1s0.network
[Match]
Name=enp1s0
[Network]
Bridge=br0
[BridgeVLAN]
VLAN=10
[BridgeVLAN]
VLAN=20
[BridgeVLAN]
VLAN=30
'''harley$''' cat 16-br0_up.network
[Match]
Name=br0
With this I get:
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 1 PVID Egress Untagged
10
20
30
br0 1 PVID Egress Untagged
'''harley$'''
But what is this? We have default VLAN <code>1 PVID Egress Untagged</code>. I don't want this. Seems setting <code>DefaultPVID=none</code> in 08-br0.netdev doesn't work. I've made a [[#Workaround for setting DefaultPVID=none|Workaround for setting DefaultPVID=none]]. Looking at this behavior I found that we can set <code>default_pvid</code> in the kernel only if <code>vlan_filtering = 0</code>. By hand I have to do:
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$''' sudo bash -c 'echo 0 >/sys/class/net/br0/bridge/default_pvid'
'''harley$''' sudo bash -c 'echo 1 >/sys/class/net/br0/bridge/vlan_filtering'
'''harley$'''
If I start a guest I will get now:
'''harley$''' virsh start deb9-test
'''harley$''' sudo bridge vlan show
port vlan ids
enp1s0 10
20
30
br0 None
vnet0 None
'''harley$'''
The virtual network interface vnet0 for deb9-test has no VLAN Id. Libvirt does not know something about this so we have to tell it. Libvirt provides [https://www.libvirt.org/hooks.html hook scripts] that we can use for this. We have to:
# [[#define VLAN-ID the virtual machine belongs to]]
# [[#get information on startup from the runtime XML-config of the domain]]
# [[#set VLAN-ID to the dynamic virtual network interface vnet*]]
For debugging the hook-scripts I've made a small script:
'''harley$''' cat debug.sh
#!/bin/bash -e
# https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.
# For debug set symlink to hook-script daemon, qemu, lxc, libxl and/or network,
# e.g. 'sudo ln -s debug.sh qemu' and restart libvirtd.

logfile='/var/log/libvirt/hooks.log'

echo "$0" >>$logfile
date -Iseconds >>$logfile
echo "\$1=$1, \$2=$2, \$3=$3, \$4=$4" >>$logfile
cat - >>$logfile
echo -e "\n---------------------------------------------" >>$logfile
'''harley$'''

=== define VLAN-ID the virtual machine belongs to ===
For thist we have an extra [https://libvirt.org/formatdomain.html#elementsMetadata element <metadata> in Domain XML format] for custom metadata. We can simply add the information to the static configuration with <code>'''harley$''' virsh edit deb9-test</code> like this (look only at the <metadata> element):
'''harley$''' virsh dumpxml deb9-test | head -n9
<domain type='kvm' id='1'>
<name>deb9-test</name>
<uuid>70d56a28-795d-4010-9403-513a4bd6b66a</uuid>
<metadata>
<my:home xmlns:my="http://hoeft-online.de/my/">
<my:vlan>10</my:vlan>
</my:home>
</metadata>
<memory unit='KiB'>1048576</memory>

=== get information on startup from the runtime XML-config of the domain ===
It seems a little bit difficult to get needed information out of the big XML-config but it's no problem with XSLT. I've made a XSL-stylesheet for this and use xmlstarlet. For developing I took a snapshot from runtime XML-config useing [[#debug.sh|debug.sh]] and prepaired it to a well formed xml-document by hand for hook-parameter $2=start. This is the result:
'''harley$''' cat qemu.xsl
<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:my="http://hoeft-online.de/my/" exclude-result-prefixes="my">
<xsl:output omit-xml-declaration="yes" indent="no"
encoding="utf-8" media-type="text/xml"/>
<xsl:strip-space elements="*"/>
<xsl:template match="text()|@*"/>

<xsl:template match="/domain">
<meta>
<xsl:apply-templates/>
</meta>
</xsl:template>

<xsl:template match="metadata/my:home/my:vlan">
<vlan>
<xsl:value-of select="."/>
</vlan>
</xsl:template>

<xsl:template match='interface[@type="bridge"]/target'>
<dev>
<xsl:value-of select="@dev"/>
</dev>
</xsl:template>

<!-- vim: set sts=2 sw=2: --&t;
</xsl:stylesheet>
'''harley$'''

'''harley$''' xmlstarlet tr qemu.xsl /var/log/libvirt/hooks.xml
<meta><vlan>10</vlan><dev>vnet0</dev></meta>'''harley$'''

=== set VLAN-ID to the dynamic virtual network interface vnet* ===
Putting it all together here is the hook-script:
'''harley$''' cat /etc/libvirt/hooks/qemu
#!/bin/bash -e
#/etc/libvirt/hooks/qemu
# Docs: https://www.libvirt.org/hooks.html
# If you make a new hook script then 'sudo systemctl restart libvirtd'.

# On startup of the domain (guest) This script does:
# get Metadata VLAN-Id of the guest and target device of the bridge from
# the domain-xml available on standard input. It is the runtime
# version from 'virsh dumpxml domainname'. For extracting the
# information we use a XSL-stylesheet. Example input into $META:
# <meta><vlan>10</vlan><dev>vnet0</dev></meta>
# Select $DEV from $META
# Select $VLAN from $META
# Set $VLAN to $DEV on the bridge

case "$2" in
prepare)
;;
start)
META=$(/usr/bin/xmlstarlet tr /etc/libvirt/hooks/qemu.xsl -)
DEV=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/dev')
VLAN=$(echo "$META" | /usr/bin/xmlstarlet sel -t -v '/meta/vlan')
if [[ -n $DEV && -n $VLAN ]]; then
/sbin/bridge vlan add vid "$VLAN" dev "$DEV"
fi
;;
started)
;;
stopped)
;;
release)
;;
migrate)
;;
restore)
;;
reconnect)
;;
attach)
;;
*)
echo "qemu hook called with unexpected options $*" >&2
exit 1
;;
esac
'''harley$'''

=== Workaround for setting DefaultPVID=none ===

=== References ===
* https://www.libvirt.org/hooks.html
* https://serverfault.com/questions/696011/libvirt-hook-qemu-suse12

[[Category:Virtualization]]

VLAN for virtual machines

2017-09-24T11:01:24Z

Ingo: linkfix

VLAN for virtual machines

2017-09-23T23:57:22Z

Ingo: revise descriptions, add discussion

VLAN for virtual machines

2017-09-22T22:05:29Z

Ingo: describe linux bridge as hub

VLAN for virtual machines

2017-09-22T14:13:39Z

Ingo: describe oldstyle linux bridge as hub

VLAN for virtual machines

2017-09-22T12:40:25Z

Ingo: describe oldstyle linux bridge as hub

VLAN for virtual machines

2017-09-22T12:17:35Z

Ingo: describe Preparation

VLAN for virtual machines

2017-09-21T22:56:34Z

Ingo: add references

VLAN for virtual machines

2017-09-21T19:55:15Z

Ingo: add references

VLAN for virtual machines

2017-09-20T23:30:26Z

Ingo: create page

Setup VGAPassthrough

2017-09-20T10:46:45Z

Ingo: create page

== HowTo Setup VGAPassthrough from scratch on Debian 9 (stretch) ==
Author: Ingo Höft, 2017-08-10

=== Preparation ===
You need a CPU with virtualization and IOMMU, two different graphic
cards, two keyboards and mice, one for the host, one for the guest. I
use one keyboard/mouse with an usb switch.

Install a basic system for testing as described in
[[Setup_KVM_with_console]] but without Guest Setup.

I have a simple older graphics card for the text console and NVIDIA for
the guest.
~$ lspci | grep VGA
0f:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
37:04.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA 1064SG [Mystique] (rev 03)
~$
The older NVIDIA Quadro 2000 does not support UEFI bios so we will
workaround using of debian package ovmf (UEFI firmware for virtual
machines). virt-install uses seabios by default. Next time I will look
for a newer graphics card supporting UEFI. If your graphics card support
UEFI you can look at [https://www.techpowerup.com/vgabios/ Video BIOS Collection]
or check with this [https://vfio.blogspot.de/2014/08/does-my-graphics-card-rom-support-efi.html rom-parser].

=== Setup VGAPassthrough on host ===
It is important to blacklist the graphics card for the guest,
here NVIDIA, so the host will never see it. This should be done
first to avoid confusion. I had trouble. My linux box does not use
the right graphics card on boot and I could not see anything. The
nouveau driver is default for NVIDIA. For blacklisting I use kernel
parameter. On grub boot menu hit 'e' and append kernel parameter
'modprobe.blacklist=nouveau' (or what your driver is) and boot with F10.

You should know what driver to blacklist before installing the second
graphics card. Maybe you can't examine it afterwards because of a black
screen. Use 'sudo lspci -v' to show what 'Kernel driver in use:' for the
video card to blacklist.

After booting I set in /etc/default/grub for persistence
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on modprobe.blacklist=nouveau"
Now I boot with the right graphics card for the console of the host.

The intel_iommu=on is needed for Intel CPUs, amd_iommu=on for AMD CPUs.
Don't forget to run
~$ update-grub
Now we have to load the vfio-pci driver modules for VGAPassthrough.
They MUST load before any other video driver so the graphics card can't
be occupied bye other drivers. We do it very early in initram, so edit
/etc/initramfs-tools/modules like this:
~$ cat /etc/initramfs-tools/modules
# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# the drivers for vfio must load early in THIS order!
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
~$

Now we have to append the physical graphics devices to the vfio-pci
driver. This is somewhat tricky. You cannot passthrough simple devices
but only complete IOMMU groups. For further information and finding the
IOMMU group look
[https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_IOMMU here]:
"An IOMMU group is the smallest set of physical devices that can be
passed to a virtual machine".

Using the small bash script from there for my NVIDA card I found:
IOMMU Group 14 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF106GL [Quadro 2000] [10de:0dd8] (rev a1)
IOMMU Group 14 0f:00.1 Audio device [0403]: NVIDIA Corporation GF106 High Definition Audio Controller [10de:0be9] (rev a1)
There are two devices in Group 14: the VGA controller and its build in
audio device. That's evident.

Now we can add the devices in the group to vfio-pci in
/etc/modprobe.d/vfio.conf like this:
~$ cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:0dd8,10de:0be9
options vfio-pci disable_vga=1
~$ sudo update-initramfs -u
You will find the device numbers from my example and I will not use VGA.
To get module parameters I use 'sudo modinfo vfio-pci'.

Reboot.

Now you should find the well set up vfio-pci driver like this:
~$ lspci -nnk -d 10de:0dd8
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF106GL [Quadro 2000] [10de:0dd8] (rev a1)
Subsystem: Hewlett-Packard Company GF106GL [Quadro 2000] [103c:084a]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
~$ lspci -nnk -d 10de:0be9
0f:00.1 Audio device [0403]: NVIDIA Corporation GF106 High Definition Audio Controller [10de:0be9] (rev a1)
Subsystem: Hewlett-Packard Company GF106 High Definition Audio Controller [103c:084a]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
~$
Inspecting 'sudo journalctl -k' is also a good thing.

=== Install virtual machine ===
Now it's up to install a guest with all this:
~$ virsh net-start default
~$ virt-install --virt-type kvm \
--cpu host \
--features kvm_hidden=on \
--name harley \
--location https://ftp.de.debian.org/debian/dists/stretch/main/installer-amd64/ \
--os-variant debian9 \
--extra-args "console=ttyS0,115200" \
--noautoconsole \
--disk size=10 \
--memory 2048 \
--host-device 0f:00.0 \
--host-device 0f:00.1 \
;
{|cellspacing="10"
| --cpu host
|Yes of course, we want hardware accelleration and this guest with its physical device binding runs only on this host.
|-
| --features kvm_hidden=on
|It was told, that Nvidia does not support running consumer-grade cards in VMs.
|-
| --noautoconsole
|virt-install returns from running guest installation, so you can easy connect with virsh. Autoconsole will not give you the guest installation screen. '--graphics none' does not work in any way although we only use text mode in guest installation. I don't know why. Maybe it has something to do with the bound graphics devices.
|-
| --host-device
|Two times - this must be the IOMMU group with its two physical devices as described above.
|}
Connect to guest installation and install as usual
~$ virsh console harley

=== Troubleshooting ===
If you get many error messages from virt-install, try to use
temporary --cpu kvm64. This is more robust and may give less and more
understandable messages.

On my machine virt-install canceled the installation with error
messages [1]. The reason was that the pc has a buggy bios and the kernel
disabled interrupt remapping (found with 'sudo journalctl -k') [2].
But IOMMU needs this and there is a way to force this feature with
iommu_unsave_interrupts like this:
~$ cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
~$ sudo update-initramfs -u
Reboot.

Of course this is also needed for machines does not support interrupt
remapping at all.

=== Conclusion ===
I wanted to setup VGAPassthrough to do my work in an efficient way. I
saw it wasn't as simple as I thought so I decided to do it from scratch
step by step to learn how it works. After spending about a week I come
to this result: I'm able to start the vm and get a login prompt on the
screen attached to the passed through graphics card. It works. But when
I restart the vm I have no output on the screen. The guests graphics
card wasn't reset properly so I have to restart the host to get it for
one time again. For me that is a showstopper but it's a known problem
and can be fixed by using UEFI. My somewhat older NVIDIA Quadro 2000
doesn't support it. I have looked for nvidia-smi but test it not
yet. I should buy a newer graphics card with UEFI support.

Next I wanted to login to the guest on it's login prompt but there is
no keybord and mouse support there. Seems that it should also passed
through but could only find generic hints in the many HowTos and manuals
like: "passthrough your keyboard and mouse to the vm, no problem". I did
it and yes, it works. But I have an USB-switch for my keyboard/mouse to
switch between host and guest. One switch-port attached to an usb-port
for the host, one switch-port attached to an usb-port for the guest.
Booting the host with nothing connected to the guests usb-port and
starting the guest with keyboard/mouse connected to guests usb-port then
I get it to the login prompt. Seems to work but when I switch to the
screen of the host without switching the usb-switch back the host will
occupy the usb-port for the guest and I loose it for the vm. Seems that
I have to hide this port on the host. When looking around for a solution
I found this [5].

That was the point I gave up. What will come up with sound? Do I
have passthrough everything? It seems there is no generic setup for
VGAPassthrough. I found many HowTows and Manuals but each only for a
special hardware. I'm not willing to investigate in development to this
and just make another HowTo for my machine. I have other projects.
Looking in a year what's going on.

=== References ===
* https://wiki.debian.org/VGAPassthrough
* https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
* https://davidyat.es/2016/09/08/gpu-passthrough/
* https://forums.linuxmint.com/viewtopic.php?f=231&t=212692
* https://vfio.blogspot.de/
* https://www.redhat.com/archives/vfio-users/
* http://www.laketide.com/setting-up-gpu-passthrough-with-kvm-on-fedora/
* https://pve.proxmox.com/wiki/Pci_passthrough#IOMMU_interrupt_remapping

=== Footnotes ===
[1] ERROR internal error: qemu unexpectedly closed the monitor:
warning: host doesn't support requested feature: CPUID.01H:EDX.ds [bit 21]
warning: host doesn't support ...

[2] Kernel message:
"DMAR-IR: This system BIOS has enabled interrupt remapping on a
chipset that contains an erratum making that feature
unstable. To maintain system stability interrupt remapping
is being disabled. Please contact your BIOS vendor for an
update"

[[Category:Virtualization]]

Setup KVM with console

2017-09-19T22:25:03Z

Ingo:

== HowTo Setup KVM only with console on Debian 9 (strech) ==
Author: Ingo Höft, 2017-08-10

I want to have a simple basic linux box with KVM but without graphics
and not needed components. So I avoid side effects on testing and
can increase its functionality step by step and use it as server
(host) for virtual machines only with command line administration as
usual for server. For this I always use 'apt install' with option
--no-install-recommends so I can decide what I realy need. I simplify
this as default for apt-get with:
echo APT::Install-Recommends \"false\"\; > /etc/apt/apt.conf.d/99install-recommends

You need a CPU with virtualization.

I use LVM so I can easy try and error with installations and revert to a
clean tested snapshot. For reference here my most used commands:
~$ sudo lvcreate -n snap -s vmhost-vg/root -L 10G #create snapshot
~$ sudo lvconvert --merge vmhost-vg/snap #revert to snapshot
~$ sudo lvremove vmhost-vg/snap #commit installation

=== Host Setup ===
Install a basic system from installation medium in textmode, e.g. from
CD-ROM or USB-Stick etc. without Desktop and any graphical environment.
That means on tasksel only select "standard system utilities".

[ ] Debian desktop environment
[ ] ... GNOME
[ ] ... Xfce
[ ] ... KDE
[ ] ... Cinnamon
[ ] ... MATE
[ ] ... LXDE
[ ] web server
[ ] print server
[ ] SSH server
[*] standard system utilities

After this we have a simple running linux box with console and no
graphics.

For my network I setup some additional features like uvesafb (if needed)
for better screen output, ssh, sssd, kerberos, nfs4. Install what you
need but no xserver-xorg.

Install KVM:
~$ sudo apt --no-install-recommends install qemu-utils qemu-kvm \
libvirt-clients dnsmasq-base ebtables libvirt-daemon-system \
virtinst libosinfo-bin firewalld

In particular this will prevent virtinst to install not needed
virt-viewer with big overhead for graphics vnc connection to client.
Check if you like with: (not needed, information only)
~$ apt --simulate install virt-viewer | less

For system wide installations
~$ export LIBVIRT_DEFAULT_URI=qemu:///system
I put this in .bashrc.

In order to be able to manage virtual machines as regular user, that
user needs to be added to some groups:
~$ sudo adduser <youruser> libvirt
~$ sudo adduser <youruser> libvirt-qemu

=== Guest Setup ===
Now install your first simple guest debian9 (stretch) with console
setup.

Before starting guest installation get rows and columns from your screen
on the host. We will need that later. And start the default network if its not running.
~$ stty --all
speed 38400 baud; rows 64; columns 160; line = 0; (example)
...
~$
~$ virsh net-start default
( may also use
~$ virsh net-autostart default )
Install the guest:
~$ virt-install --virt-type kvm \
--cpu kvm64 \
--name base \
--location https://ftp.de.debian.org/debian/dists/stretch/main/installer-amd64/ \
--os-variant debian9 \
--extra-args "console=ttyS0" \
--graphics none \
--disk size=4 \
--memory 512 \
;
You may use '--cpu host' for better performance, but may cause issues if
migrating the guest to a host without an identical CPU. Show supported
CPU models with:
~$ virsh cpu-models x86_64

The installation program of the guest should come up in text mode now
and you can complete installation on the guest as usual. I also install
only 'standard system utilities' for testing. On reboot on grub boot
menu hit 'e', append kernel parameter 'console=ttyS0' and boot
with F10.

Login to guest and first set
~$ stty rows 64 columns 160 (or what yours is)
~$

Then next set /etc/default/grub to
GRUB_CMDLINE_LINUX_DEFAULT="quiet console=ttyS0"
Don't forget update-grub.

=== References ===
* https://www.debian.org/distrib/
* https://wiki.debian.org/KVM
[[Category:Virtualization]]

Kategorie:Virtualization

2017-09-19T22:24:32Z

Ingo: create category

Everything to Virtualization technology

Hauptseite

2017-09-19T21:35:58Z

Ingo: Redirect to Categories

#REDIRECT [[Spezial:Kategorien]]

Hauptseite

2017-09-19T21:12:59Z

Ingo: redirect to main page

#REDIRECT [[main page]]