VLAN for virtual machines: Unterschied zwischen den Versionen
Ingo (Diskussion | Beiträge) (describe linux bridge as hub) |
Ingo (Diskussion | Beiträge) (revise descriptions, add discussion) |
||
Zeile 2: | Zeile 2: | ||
I wanted to update VLAN connections for virtual | I wanted to update VLAN connections for virtual | ||
machines to newer technologies and put a question on | machines to newer technologies and put a question on | ||
− | [https://unix.stackexchange.com/questions/392758/setup-vlan-on-linux-bridge-for-virtual-machines-with-systemd | + | [https://unix.stackexchange.com/questions/392758/setup-vlan-on-linux-bridge-for-virtual-machines-with-systemd unix.stackexchange]. But I do not get any answer. It seems there is very |
little knowledge for this out there. So I decided to work on it by myself | little knowledge for this out there. So I decided to work on it by myself | ||
and document it here. | and document it here. | ||
Zeile 13: | Zeile 13: | ||
== Preparation == | == Preparation == | ||
− | I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing | + | I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: [[Setup KVM with console]]. I'm sitting on harley as host, my all day workstation. |
Now I start the virtual machine, login and show its interface setting: | Now I start the virtual machine, login and show its interface setting: | ||
'''harley$''' virsh start --console deb9-test | '''harley$''' virsh start --console deb9-test | ||
Zeile 32: | Zeile 32: | ||
Name=vlan10 | Name=vlan10 | ||
[Network] | [Network] | ||
− | + | DHCP=ipv4 | |
− | + | IPv6AcceptRA=no | |
− | + | LinkLocalAddressing=no | |
− | '''deb9-test$''' | + | To test if the virtual machine has connection I use: |
+ | '''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service | ||
+ | -- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. -- | ||
+ | Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed | ||
+ | Sep 23 20:34:05 systemd[1]: Started Network Service. | ||
+ | Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready | ||
+ | Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success | ||
+ | Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier | ||
+ | Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier | ||
+ | Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL | ||
+ | Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL | ||
+ | Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1 | ||
+ | Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured | ||
+ | Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured | ||
+ | '''deb9-test$''' | ||
+ | 4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this: | ||
+ | '''deb9-test$''' journalctl -b --no-hostname -u systemd-networkd.service | ||
+ | -- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. -- | ||
+ | Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed | ||
+ | Sep 23 20:44:59 systemd[1]: Started Network Service. | ||
+ | Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready | ||
+ | Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success | ||
+ | Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier | ||
+ | Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier | ||
+ | Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL | ||
+ | Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL | ||
+ | Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured | ||
+ | '''deb9-test$''' | ||
+ | |||
Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest. | Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest. | ||
'''deb9-test$''' grep ^ExecStart= /lib/systemd/system/serial-getty@.service | '''deb9-test$''' grep ^ExecStart= /lib/systemd/system/serial-getty@.service | ||
Zeile 41: | Zeile 69: | ||
modify to | modify to | ||
ExecStart=-/sbin/agetty --autologin ''yourloginname'' --keep-baud 115200,38400,9600 %I $TERM | ExecStart=-/sbin/agetty --autologin ''yourloginname'' --keep-baud 115200,38400,9600 %I $TERM | ||
+ | To list all settings of the bridge you can use: | ||
+ | '''harley$''' find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort | ||
== oldstyle linux bridge as hub == | == oldstyle linux bridge as hub == | ||
Zeile 76: | Zeile 106: | ||
0 | 0 | ||
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets. | Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets. | ||
− | |||
− | |||
=== References === | === References === | ||
Zeile 112: | Zeile 140: | ||
[Match] | [Match] | ||
Name=br0 | Name=br0 | ||
+ | '''harley$''' sudo ip link set dev br0 down && sudo ip link del dev br0 | ||
'''harley$''' sudo systemctl restart systemd-networkd | '''harley$''' sudo systemctl restart systemd-networkd | ||
'''harley$''' | '''harley$''' | ||
− | + | AgeingTimeSec=0 is not acepted but should: | |
'''harley$''' cat /sys/class/net/br0/bridge/ageing_time | '''harley$''' cat /sys/class/net/br0/bridge/ageing_time | ||
30000 (means 300 sec) | 30000 (means 300 sec) | ||
+ | '''harley$''' | ||
+ | But I've found a workaround. Useing a number between '''.'''01 and '''.'''000001 (there are dots) will set ageing_time to 0. | ||
+ | So set AgeingTimeSec='''.'''000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we | ||
+ | will get: | ||
+ | '''harley$''' cat /sys/class/net/br0/bridge/ageing_time | ||
+ | 0 | ||
'''harley$''' cat /sys/class/net/br0/bridge/stp_state | '''harley$''' cat /sys/class/net/br0/bridge/stp_state | ||
0 | 0 | ||
Zeile 123: | Zeile 158: | ||
0 | 0 | ||
'''harley$''' | '''harley$''' | ||
+ | The guest gets now an IP-Address on boot and is connected to VLAN 10. | ||
+ | |||
+ | === Discussion === | ||
+ | This works because of three conditions. | ||
+ | # ageing time is 0: ageing time specifies the number of seconds a MAC Address will be kept in the forwarding database after having a packet received from this MAC Address. Setting it to 0 means there is never a MAC Address stored in the FDB. | ||
+ | # unicast flood on interfaces is on: this controls whether the bridge should flood traffic for which an FDB entry is missing and the destination is unknown through this port. Defaults to on. | ||
+ | # spanning tree protocol (stp) is disabled: we don't have a forward_delay at startup for the learning phase of spanning tree. | ||
+ | I have a running and connected virtual machine: | ||
+ | '''harley$''' sudo bridge vlan show | ||
+ | port vlan ids | ||
+ | enp1s0 1 PVID Egress Untagged | ||
+ | br0 1 PVID Egress Untagged | ||
+ | vnet0 1 PVID Egress Untagged | ||
+ | '''harley$''' cat /sys/class/net/br0/bridge/ageing_time | ||
+ | 0 | ||
+ | '''harley$''' cat /sys/class/net/br0/bridge/forward_delay | ||
+ | 1500 | ||
+ | '''harley$''' cat /sys/class/net/br0/bridge/stp_state | ||
+ | 0 | ||
+ | Indeed we have forward_delay 1500 (means 15 sec) but it doesn't matter. stp_state is 0 (disabled), no spanning tree. Flood (means unicast flood) is on as I can see: | ||
+ | '''harley$''' sudo bridge -d link show | ||
+ | ''3: enp1s0'' state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4 | ||
+ | hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on | ||
+ | ''95: vnet0'' state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 | ||
+ | hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on | ||
+ | '''harley$''' | ||
+ | |||
+ | |||
+ | Let's have a look at flooding on the interfaces. I disable it on the physical interface enp1s0 of the bridge and reboot the guest: | ||
+ | '''harley$''' sudo bridge link set dev enp1s0 flood off | ||
+ | '''harley$''' | ||
+ | The guest gets an IP-Address from the DHCP-Server but then can't ping its gateway. DHCP-REQUEST is broadcast and goes thru enp1s0. DHCP-ANSWER comes back thru it to any other (here only vnet0) interface which has flood on. Ping is unicast and isn't forwareded on enp1s0. If I set enp1s0 flood on and vnet0 flood off and <code>'''deb9-test$ '''sudo systemctl restart systemd-networkd</code>, I get no IP-Address from DHCP-Server and can't ping the interface. Incoming DHCP-ANSWER isn't broadcast and vnet0 doesn't forward it to the guest. | ||
+ | |||
+ | Btw. this method has bad performance as we can see with monitor. We insert MAC-Addresses into FDB for just deleting it immediately, all for nothing. | ||
+ | '''harley$''' sudo bridge monitor fdb | ||
+ | 52:54:00:01:76:20 dev enp1s0 master br0 | ||
+ | 52:54:00:b0:ca:63 dev vnet0 master br0 | ||
+ | f4:f2:6d:2c:87:f7 dev enp1s0 master br0 | ||
+ | 00:80:3f:2a:31:1a dev enp1s0 master br0 | ||
+ | Deleted 52:54:00:01:76:20 dev enp1s0 master br0 stale | ||
+ | Deleted 52:54:00:b0:ca:63 dev vnet0 master br0 stale | ||
+ | Deleted 00:80:3f:2a:31:1a dev enp1s0 master br0 stale | ||
+ | Deleted f4:f2:6d:2c:87:f7 dev enp1s0 master br0 stale | ||
+ | ... | ||
=== References === | === References === |
Version vom 24. September 2017, 01:57 Uhr
Inhaltsverzeichnis
Introduction
I wanted to update VLAN connections for virtual machines to newer technologies and put a question on unix.stackexchange. But I do not get any answer. It seems there is very little knowledge for this out there. So I decided to work on it by myself and document it here.
In gerneral I will look at four methods:
- oldstyle linux bridge as hub
- linux bridge as hub
- linux bridge with libvirt hook scripts
- Open vSwitch
Preparation
I have Debian GNU/Linux 9.1 (stretch) on the host and on virtual machines for testing. Setup is described here: Setup KVM with console. I'm sitting on harley as host, my all day workstation. Now I start the virtual machine, login and show its interface setting:
harley$ virsh start --console deb9-test
login
deb9-test$ cat /etc/systemd/network/08-vlan10.netdev [NetDev] Name=vlan10 Kind=vlan [VLAN] Id=10 deb9-test$ cat /etc/systemd/network/12-vlan10_attach-to-if.network [Match] Name=ens2 [Network] VLAN=vlan10 deb9-test$ cat /etc/systemd/network/16-vlan10_up.network [Match] Name=vlan10 [Network] DHCP=ipv4 IPv6AcceptRA=no LinkLocalAddressing=no
To test if the virtual machine has connection I use:
deb9-test$ journalctl -b --no-hostname -u systemd-networkd.service -- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:34:20 CEST. -- Sep 23 20:34:05 systemd-networkd[204]: Enumeration completed Sep 23 20:34:05 systemd[1]: Started Network Service. Sep 23 20:34:05 systemd-networkd[204]: vlan10: netdev ready Sep 23 20:34:05 systemd-networkd[204]: ens2: IPv6 enabled for interface: Success Sep 23 20:34:05 systemd-networkd[204]: ens2: Gained carrier Sep 23 20:34:05 systemd-networkd[204]: vlan10: Gained carrier Sep 23 20:34:06 systemd-networkd[204]: ens2: Gained IPv6LL Sep 23 20:34:06 systemd-networkd[204]: vlan10: Gained IPv6LL Sep 23 20:34:09 systemd-networkd[204]: vlan10: DHCPv4 address 192.168.10.89/24 via 192.168.10.1 Sep 23 20:34:09 systemd-networkd[204]: vlan10: Configured Sep 23 20:34:19 systemd-networkd[204]: ens2: Configured deb9-test$
4 sec after Started Network Service it gets an IP-Address and 14 sec later interface ens2 was Configured. If ens2 is Configured and the guest hasn't got an IP-Address the connection failed. It looks like this:
deb9-test$ journalctl -b --no-hostname -u systemd-networkd.service -- Logs begin at Fri 2017-09-15 17:09:51 CEST, end at Sat 2017-09-23 20:45:13 CEST. -- Sep 23 20:44:59 systemd-networkd[197]: Enumeration completed Sep 23 20:44:59 systemd[1]: Started Network Service. Sep 23 20:44:59 systemd-networkd[197]: vlan10: netdev ready Sep 23 20:44:59 systemd-networkd[197]: ens2: IPv6 enabled for interface: Success Sep 23 20:44:59 systemd-networkd[197]: ens2: Gained carrier Sep 23 20:44:59 systemd-networkd[197]: vlan10: Gained carrier Sep 23 20:45:00 systemd-networkd[197]: ens2: Gained IPv6LL Sep 23 20:45:00 systemd-networkd[197]: vlan10: Gained IPv6LL Sep 23 20:45:13 systemd-networkd[197]: ens2: Configured deb9-test$
Because I have to start the test virtual machine many times I setup autologin. It's no problem. There is nothing on the guest.
deb9-test$ grep ^ExecStart= /lib/systemd/system/serial-getty@.service ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM
modify to
ExecStart=-/sbin/agetty --autologin yourloginname --keep-baud 115200,38400,9600 %I $TERM
To list all settings of the bridge you can use:
harley$ find /sys/class/net/br0/bridge/ -type f -readable -printf '%f = ' -exec cat {} \; | sort
oldstyle linux bridge as hub
This works always with the old linux bridge that do not know anything about VLAN. The trick is to set it to a complete transparent state for all connected interfaces like a hub. But you have to know that the bridge will then forward all packets to all interfaces simultanously. You can do it by setting the ageing time to 0.
Disable systemd-networkd and start networking with ifupdown:
harley$ sudo systemctl stop systemd-networkd Warning: Stopping systemd-networkd.service, but it can still be activated by: systemd-networkd.socket harley$ sudo systemctl disable systemd-networkd Removed /etc/systemd/system/multi-user.target.wants/systemd-networkd.service. Removed /etc/systemd/system/sockets.target.wants/systemd-networkd.socket. harley$ sudo ip link set dev br0 down && sudo ip link del dev br0 harley$ sudo systemctl enable networking.service Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable networking harley$ sudo systemctl start networking.service harley$
Setup the bridge and start it:
harley$ cat /etc/network/interfaces auto br0 iface br0 inet manual bridge_ports enp1s0 bridge_ageing 0 bridge_stp off harley$ sudo ifup br0 Waiting for br0 to get ready (MAXWAIT is 32 seconds). harley$
It's all in place now:
harley$ cat /sys/class/net/br0/bridge/ageing_time 0 harley$ cat /sys/class/net/br0/bridge/stp_state 0 harley$ cat /sys/class/net/br0/bridge/vlan_filtering 0
Yes, there is no VLAN filtering, means VLAN on the bridge is disabled but the guest sees the VLAN-tagged packets.
References
linux bridge as hub
Now I try to setup #oldstyle linux bridge as hub just with systemd-networkd.
Disable networking with ifupdown and start systemd-networkd:
harley$ sudo systemctl stop networking.service harley$ sudo systemctl disable networking.service Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install disable networking harley$ sudo ip link set dev br0 down && sudo ip link del dev br0 harley$ sudo systemctl enable systemd-networkd Created symlink /etc/systemd/system/multi-user.target.wants/systemd-networkd.service → /lib/systemd/system/systemd-networkd.service. Created symlink /etc/systemd/system/sockets.target.wants/systemd-networkd.socket → /lib/systemd/system/systemd-networkd.socket. harley$ sudo systemctl start systemd-networkd harley$
Setup the bridge and start it:
harley$ cat /etc/systemd/network/08-br0.netdev [NetDev] Name=br0 Kind=bridge [Bridge] AgeingTimeSec=0 STP=false harley$ cat /etc/systemd/network/12-br0_add-enp1s0.network [Match] Name=enp1s0 [Network] Bridge=br0 harley$ cat /etc/systemd/network/16-br0_up.network [Match] Name=br0 harley$ sudo ip link set dev br0 down && sudo ip link del dev br0 harley$ sudo systemctl restart systemd-networkd harley$
AgeingTimeSec=0 is not acepted but should:
harley$ cat /sys/class/net/br0/bridge/ageing_time 30000 (means 300 sec) harley$
But I've found a workaround. Useing a number between .01 and .000001 (there are dots) will set ageing_time to 0. So set AgeingTimeSec=.000001 in /etc/systemd/network/08-br0.netdev. I suppose it's a bug. Then we will get:
harley$ cat /sys/class/net/br0/bridge/ageing_time 0 harley$ cat /sys/class/net/br0/bridge/stp_state 0 harley$ cat /sys/class/net/br0/bridge/vlan_filtering 0 harley$
The guest gets now an IP-Address on boot and is connected to VLAN 10.
Discussion
This works because of three conditions.
- ageing time is 0: ageing time specifies the number of seconds a MAC Address will be kept in the forwarding database after having a packet received from this MAC Address. Setting it to 0 means there is never a MAC Address stored in the FDB.
- unicast flood on interfaces is on: this controls whether the bridge should flood traffic for which an FDB entry is missing and the destination is unknown through this port. Defaults to on.
- spanning tree protocol (stp) is disabled: we don't have a forward_delay at startup for the learning phase of spanning tree.
I have a running and connected virtual machine:
harley$ sudo bridge vlan show port vlan ids enp1s0 1 PVID Egress Untagged br0 1 PVID Egress Untagged vnet0 1 PVID Egress Untagged harley$ cat /sys/class/net/br0/bridge/ageing_time 0 harley$ cat /sys/class/net/br0/bridge/forward_delay 1500 harley$ cat /sys/class/net/br0/bridge/stp_state 0
Indeed we have forward_delay 1500 (means 15 sec) but it doesn't matter. stp_state is 0 (disabled), no spanning tree. Flood (means unicast flood) is on as I can see:
harley$ sudo bridge -d link show 3: enp1s0 state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on 95: vnet0 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on mcast_flood on harley$
Let's have a look at flooding on the interfaces. I disable it on the physical interface enp1s0 of the bridge and reboot the guest:
harley$ sudo bridge link set dev enp1s0 flood off harley$
The guest gets an IP-Address from the DHCP-Server but then can't ping its gateway. DHCP-REQUEST is broadcast and goes thru enp1s0. DHCP-ANSWER comes back thru it to any other (here only vnet0) interface which has flood on. Ping is unicast and isn't forwareded on enp1s0. If I set enp1s0 flood on and vnet0 flood off and deb9-test$ sudo systemctl restart systemd-networkd
, I get no IP-Address from DHCP-Server and can't ping the interface. Incoming DHCP-ANSWER isn't broadcast and vnet0 doesn't forward it to the guest.
Btw. this method has bad performance as we can see with monitor. We insert MAC-Addresses into FDB for just deleting it immediately, all for nothing.
harley$ sudo bridge monitor fdb 52:54:00:01:76:20 dev enp1s0 master br0 52:54:00:b0:ca:63 dev vnet0 master br0 f4:f2:6d:2c:87:f7 dev enp1s0 master br0 00:80:3f:2a:31:1a dev enp1s0 master br0 Deleted 52:54:00:01:76:20 dev enp1s0 master br0 stale Deleted 52:54:00:b0:ca:63 dev vnet0 master br0 stale Deleted 00:80:3f:2a:31:1a dev enp1s0 master br0 stale Deleted f4:f2:6d:2c:87:f7 dev enp1s0 master br0 stale ...
References
- https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
- https://www.freedesktop.org/software/systemd/man/systemd.network.html
linux bridge with libvirt hook scripts
References
- https://www.libvirt.org/hooks.html
- https://serverfault.com/questions/696011/libvirt-hook-qemu-suse12