I’ve been meaning to write a post on ethernet bridges and how they can easily be used to accommodate virtual machines with their “own” physical NIC(Network Interface Card). I see a ton of post online about people struggling to get multiple ethernet cards to work in Xen and other hypervisors like KVM. A common complaint is that when both NIC cards are plugged into the LAN they lose connectivity from all machines including the host.
Many sites make an attempt to explain the problem of multiple interfaces on the same network by walking you through a Xen custom configuration. However they fail to identify the concept of bridges, layer 2 loops and why Spanning Tree Protocol is your friend! So many virtualization nuts(like myself) spend hours trying to find a problem with Xen, Vmware, KVM, whatever… when the problem may just be how the interfaces are configured.
The goals of this post:
* define ethernet bridging
* explain ethernet loops
* discuss how this relates to VM’s and the hypervisor
* LAB: set up two ethernet cards for guest VM and my Fedora 10 KVM Server
An ethernet bridge is one where separate network interfaces are bonded together for the purpose of passing ethernet frames to another logical section of the network. An excellent example of this are 802.11x wireless bridges. Many name brand wifi routers support this mode of setup. Each end of the wireless router is pointed at the other(with a directional antenna).The routers each have two interfaces with two separate MAC addresses. Each side know’s the other sides WiFi MAC address. In this scenario all traffic that is destined to the WiFi network gets forwarded to the other side.
Virtual Machines can operate in the same fashion. eth0 might be the Virtual Server’s NIC and eth1 can be dedicated to the Virtual machines. This is accomplished by creating a dummy interface to create a virtual bridge. It works in an identical manor to the wireless networking product. eth1 will just listen and pass all ethernet frames to virtual interface br1(bridge 1). Bridge br1 will act like a virtual ethernet switch for VM’s to virtually “plug into”.
Note: The setup below will not work without Spanning Tree Protocol (STP) enabled on one of the links. More on that further down.
Layer 2 loops
A layer 2 loop occurs when multiple switches are connected together from multiple interfaces. What happens is that MAC address tables are being generated on multiple paths to the interconnected switches. The switches have no idea where to send the ethernet frames. So as a general rule a loop cannot exist in a switched network. The exception to this is with the use of a handy layer 2 protocol known as ‘Spanning Tree Protocol’ (STP).
The Wikipedia has a good description of what STP is:
“he Spanning Tree Protocol is an OSI layer-2 protocol that ensures a loop-free topology for any bridged LAN. It is based on an algorithm invented by Radia Perlman while working for Digital Equipment Corporation. Spanning tree allows a network design to include spare (redundant) links to provide automatic backup paths if an active link fails, without the danger of bridge loops, or the need for manual enabling/disabling of these backup links. Bridge loops must be avoided because they result in flooding the network.
The Spanning Tree Protocol (STP), is defined in the IEEE Standard 802.1D. As the name suggests, it creates a spanning tree within a mesh network of connected layer-2 bridges (typically Ethernet switches), and disables those links that are not part of the tree, leaving a single active path between any two network nodes.”
How this Relates to Virtual Macinies
So the question is what does wireless bridges, layer 2 loops, and Spanning Tree Protocol have to do with using two interfaces on Xen? What I’m trying to demonstrate is that ethernet bridges real or virtual are subject to layer 2 loops. If you plug two configured ethernet cards into a switch you will create a loop that will kill both connections. Over and over again in online forums this befuddles people. They spend hours trying to fix the virtual server, but in fact it is a layer 2 loop that is stopping them. Many help articles try to explain this but usually they skip to the fix. To eliminate loops you must enable STP on one of the links.
Lab: Setup two NIC’s on KVM
This lab is for setting up the two NIC’s; for the Virtual Server and Guest VM respectivly. Both ethernet cards will be connected to the same switch. We will use the Linux ‘brctl’ command to create a virtual bridge implemented with STP. For the record, I’m using Fedora 10, KVM, QEMU, libvirt and virt-manager on standard i386 yada, yada…
Step 1 Configure the Interfaces
We will start by removing Network Manager. It’s a horrible tool that will mess with are custom interface scripts.
Now we will edit the eth1 to bridge to br1. Please note you can use any interface you want by renaming them appropriately. On a Fedora/Red Hat system interface configuration files are stored in ‘/etc/sysconfig/network-scripts/’.
Open you favorite editor and create the virtual bridge interface br1. The file should be named ‘/etc/sysconfig/network-scripts/ifcfg-br1’
Now we edit eth1 to bind it to the br1.
Restart the network interfaces. br1 will time out, which is okay because we haven’t started STP yet.
Now that we have bridge interface br1 paired with eth1, we can use the ‘brctl’ command to bind them in the kernel. brctl also is used to set STP on a bridge interface.
now enable STP.
Now lets restart the network and set our Virtual Machine with virt-manager to br1!
If you are experiencing kernel panics since setting up the Bridge CLICK HERE.