Guests are offline

This is a very broad title, and will probably not help anyone except for me if this happens for me again.

One day when I had to restart my KVM host, none of my guests had internet connection. The eth0 on the KVM host was attached to a bridge (br0) and that host had connectivity through the bridge. Here are some configs (which had not changed since it worked):

# /etc/network/interfaces

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet manual
#iface eth0 inet dhcp


auto br0
iface br0 inet dhcp
	bridge_ports eth0
	bridge_stp off
	bridge_fd 0
	bridge_maxwait 0

Configuration seems good to me. I have never needed stp=on, so it should be alright.

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether (...)
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether (...)
    inet 10.0.0.107/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::baae:edff:feea:92ce/64 scope link
       valid_lft forever preferred_lft forever
(...)
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether (...)
    inet6 fe80::fc54:ff:feb8:4296/64 scope link
       valid_lft forever preferred_lft forever
(...)
9: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether (...)
    inet6 fe80::fc54:ff:fec8:f2e/64 scope link
       valid_lft forever preferred_lft forever
10: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether (...)
    inet6 fe80::fc54:ff:feaf:d5b9/64 scope link
       valid_lft forever preferred_lft forever
(...)
13: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether (...)
    inet6 fe80::fc54:ff:feed:6dc4/64 scope link
       valid_lft forever preferred_lft forever

Everything seems in order. We have an ip address on br0 and everything seems attached to the bridge.

$ sudo brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.b8aeedea92ce	no		eth0
							vnet0
							vnet1
							vnet2
							vnet3

Some duplicate info here, but just to make sure the bridge is set up correctly.

Then i saw that iptables had the following policy:

Chain FORWARD (policy DROP 8254 packets, 2386K bytes)

Change chain policy to make sure forwards are sent properly.

sudo iptables --policy FORWARD ACCEPT

I am unsure why it happened in the first place. Maybe related to docker in some way? Since much of my iptables configuration is created by services I am unsure how to persist this in a good way. So I probably will need to fix this next time I reboot (which will likely be in several years), and by then I will have forgotten about it. Hence this post.

UPDATE: Apparently it is a race condition between kvm and docker. Depending on who starts first. I only have to apply this about 50% of the time.