Feb
26
2009

Using Multiple interfaces with KVM and Xen

I’ve been meaning to write a post on ethernet bridges and how they can easily be used to accommodate virtual machines with their “own” physical NIC(Network Interface Card).  I see a ton of post online about people struggling to get multiple ethernet cards to work in Xen and other hypervisors like KVM.  A common complaint is that when both NIC cards are plugged into the LAN they lose connectivity from all machines including the host.

Many sites make an attempt to explain the problem of multiple interfaces on the same network by walking you through a Xen custom configuration.  However they fail to identify the concept of bridges, layer 2 loops and why Spanning Tree Protocol is your friend!  So many virtualization nuts(like myself) spend hours trying to find a problem with Xen, Vmware, KVM, whatever… when the problem may just be how the interfaces are configured.

The goals of this post:

* define ethernet bridging

* explain ethernet loops

* discuss how this relates to VM’s and the hypervisor

* LAB: set up two ethernet cards for guest VM and my Fedora 10 KVM Server

Ethernet Bridges

An ethernet bridge is one where separate network interfaces are bonded together for the purpose of passing ethernet frames to another logical section of the network.  An excellent example of this are 802.11x wireless bridges.  Many name brand wifi routers support this mode of setup.  Each end of the wireless router is pointed at the other(with a directional antenna).The routers each have two interfaces with two separate MAC addresses.  Each side know’s the other sides WiFi MAC address.  In this scenario all traffic that is destined to the WiFi network gets forwarded to the other side.

Virtual Machines can operate in the same fashion.  eth0 might be the Virtual Server’s NIC and eth1 can be dedicated to the Virtual machines.  This is accomplished by creating a dummy interface to create a virtual bridge.  It works in an identical manor to the wireless networking product.  eth1 will just listen and pass all ethernet frames to virtual interface br1(bridge 1).  Bridge br1 will act like a virtual ethernet switch for VM’s to virtually “plug into”.

Note:  The setup below will not work without Spanning Tree Protocol (STP) enabled on one of the links.  More on that further down.

Layer 2 loops

A layer 2 loop occurs when multiple switches are connected together from multiple interfaces.  What happens is that MAC address tables are being generated on multiple paths to the interconnected switches.  The switches have no idea where to send the ethernet frames.  So as a general rule a loop cannot exist in a switched network.  The exception to this is with the use of a handy layer 2 protocol known as ‘Spanning Tree Protocol’ (STP).

The Wikipedia has a good description of what STP is:

“he Spanning Tree Protocol is an OSI layer-2 protocol that ensures a loop-free topology for any bridged LAN. It is based on an algorithm invented by Radia Perlman while working for Digital Equipment Corporation[1][2]. Spanning tree allows a network design to include spare (redundant) links to provide automatic backup paths if an active link fails, without the danger of bridge loops, or the need for manual enabling/disabling of these backup links. Bridge loops must be avoided because they result in flooding the network.

The Spanning Tree Protocol (STP), is defined in the IEEE Standard 802.1D. As the name suggests, it creates a spanning tree within a mesh network of connected layer-2 bridges (typically Ethernet switches), and disables those links that are not part of the tree, leaving a single active path between any two network nodes.”

http://en.wikipedia.org/wiki/Spanning_tree_protocol

How this Relates to Virtual Macinies

So the question is what does wireless bridges, layer 2 loops, and Spanning Tree Protocol have to do with using two interfaces on Xen?  What I’m trying to demonstrate is that ethernet bridges real or virtual are subject to layer 2 loops.  If you plug two configured ethernet cards into a switch you will create a loop that will kill both connections.  Over and over again in online forums this befuddles people.  They spend hours trying to fix the virtual server, but in fact it is a layer 2 loop that is stopping them.  Many help articles try to explain this but usually they skip to the fix.  To eliminate loops you must enable STP on one of the links.

Lab: Setup two NIC’s on KVM

This lab is for setting up the two NIC’s; for the Virtual Server and Guest VM respectivly.  Both ethernet cards will be connected to the same switch.  We will use the Linux ‘brctl’ command to create a virtual bridge implemented with STP.  For the record, I’m using Fedora 10, KVM, QEMU, libvirt and virt-manager on standard i386 yada, yada…

Step 1  Configure the Interfaces

We will start by removing Network Manager.  It’s a horrible tool that will mess with are custom interface scripts.

[root@mattcom1 ~]# yum -y remove NetworkManager

Now we will edit the eth1 to bridge to br1.  Please note you can use any interface you want by renaming them appropriately.  On a Fedora/Red Hat system interface configuration files are stored in ‘/etc/sysconfig/network-scripts/’.

Open you favorite editor and create the virtual bridge interface br1.  The file should be named ‘/etc/sysconfig/network-scripts/ifcfg-br1’

ifcfg-br1:

DEVICE=br1
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes

Now we edit eth1 to bind it to the br1.

ifcfg-eth1:

TYPE=Ethernet
DEVICE=eth1
HWADDR=00:0a:5e:45:7e:ca  # <—- this should be your NIC’s MAC address
ONBOOT=yes
USERCTL=no
PEERDNS=yes
IPV6INIT=no
bridge=br1                                # <—- this value points to the virtual bridge

Restart the network interfaces.  br1 will time out, which is okay because we haven’t started STP yet.

[root@mattcom1 ~]# service network restart
Shutting down interface br1:                               [  OK  ]
Shutting down interface eth0_rename:                       [  OK  ]
Shutting down interface eth1:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0_rename:                         [  OK  ]
Bringing up interface eth1:                                [  OK  ]
Bringing up interface br1:
Determining IP information for br1…

Now that we have bridge interface br1 paired with eth1, we can use the ‘brctl’ command to bind them in the kernel.  brctl also is used to set STP on a bridge interface.

[root@mattcom1 ~]# brctl addif br1 eth1

now enable STP.

[root@mattcom1 ~]# brctl stp br1 on

[root@mattcom1 ~]# brctl show
bridge name    bridge id        STP enabled    interfaces
br1        8000.000a5e457eca    yes        eth1
pan0        8000.000000000000    no

Now lets restart the network and set our Virtual Machine with virt-manager to br1!

UPDATE:

If you are experiencing kernel panics since setting up the Bridge CLICK HERE.





20 Comments »

  • Nuno Ferreira

    Hi

    Very good article. I was a bit desperate to understand how bridges work, and I can say that already learned something more.

    Nevertheless, after a reboot to the system, it seems to me that the connection between the interfaces and the bridge is lost, right? As well as the STP.

    Cheers,
    Nuno

    Comment | April 2, 2009
  • mattb

    Hi Nuno,

    You are correct, what I did is create a BASH script that loads these settings on boot. There are many ways to do this, but the simple way is to add a line to your ‘/etc/rc.d/rc.local’. From this file you can automatically execute any final commands during bootup. Below is an example script I wrote for such a purpose. Keep in mind my scripting skills are not very good! I hope this helps, thank you for commenting!
    =======================================================
    #!/bin/bash

    brctl addif br2 eth2
    brctl stp br2 on
    brctl show
    service libvirtd restart
    exit
    =======================================================

    Comment | April 2, 2009
  • David Cartwright

    Two questions:

    Based on your ifcfg-eth1 and ifcfg-br1 config files, is “brctl addif br1 eth1” actually necessary? I suspect not.

    I haven’t tested this, but wondering if you could also have specified “STP=on” in the bridge config file that would remove the need to run “brctl stp br1 on”.

    Since you already have the ifcfg-eth1 and ifcfg-br1 config files, I think it would be neater to specify everything using those config files.

    btw … thanks for the tip on the use of STP … has been a timely pointer that will help me in an upcoming application deployment.

    cheers. .. .david

    Comment | April 24, 2009
  • Dave,

    First off, I’d like to say thank you for my visiting my site. I wasn’t aware that I could specify STP=on in the /etc/sysconfig/network-scripts/ifcfg-eth*. I’m going to give that try; thanks for the tip.

    One thing I have to warn you about. There is a bug in the bridge utils for Fedora and as I understand it Unbuntu as well. Very occasionly the whole Linux system will lock up. This may be due to DHCP request by virtual host over the virtual bridge. Also logs will reveal ACPI errors. Although, I’ve tried all ACPI settings in my BIOS and in Linux but with no success.

    I never experienced this problem in Fedora 8/Xen, but Fedora 9 & 10/KVM this happens every couple of days the system is up. Please be careful with your productions servers. And PLEASE keep reading http://www.savelono.com. Your friend,

    -Matt

    Comment | April 24, 2009
  • David Cartwright

    Hi Matt

    a couple of things:

    1. I have confirmed that STP=on indeed works when you put it in the config file. [Fedora 10]

    2. I am using KVM on a number of production systems and haven’t seen any lock-up problems. Albeit all of the production VMs are using static IPs.

    3. There are a few extra tricks that might be useful regarding iptables etc that I have listed in my 24 Sept 2008 post here:
    http://www.linux-kvm.com/content/using-bridged-networking-virt-manager

    4. I am just in the process of bringing up a system that incorporates a bonded bridged interface that appears to be working well in testing. In some situations bonding will be advantageous vs multiple IP addresses with STP since fail-over can be automatic.

    cheers …. david

    Comment | May 7, 2009
  • mattb

    Hi Dave,

    You got my attention, what is a ‘bonded bridged interface’?

    -Matt

    Comment | May 7, 2009
  • David Cartwright

    Matt

    There’s a good overview of Bonding here:
    http://www.linuxfoundation.org/en/Net:Bonding

    With Bonding + Bridging, assume two interfaces eth0 and eth1. Bonding combines the two of them into one virtual interface, e.g. bond0 that has a single IP address.

    The bond0 interface can also be configured as a bridge, so that Virtual Machines can use it in exactly the same way as they use an un-bonded bridge.

    I haven’t tested a bonded bridge with the mode 0 load-balancing option, but it works fine in active-backup mode 1. Since eth0 and eth1 can be plugged into different switches, it also provides automatic failover at the switch level.

    A pretty good option I think!

    cheers …. david

    Comment | May 11, 2009
  • Sean

    This was extremely helpful.

    I spent mind-boggling amounts of time trying to get my setup to work properly with my VMs and the host machine to be properly accessible on the network but everything I read always pointed me towards the wrong thing (and a lot of times STP off) or gave me too little information.. Honestly, this really helps me out.

    Thanks so much,
    Sean

    Comment | October 22, 2009
  • mattb

    Hey Sean,

    Thanks for reading, were you able to assign interfaces to guest VM’s eventually? If not, maybe I can help further. Let me know!

    -Matt

    Comment | October 23, 2009
  • Sean

    Matt,

    I had no problem assigning interfaces. The correct bridge settings with multiple interfaces happened to be the biggest hurdle for me because of the lack of consistent documentation, explanation, and I think to some extent understanding amongst different users and distros. I honestly lost a lot of time to this issue to the point that I eventually just put it on the back burner for about 2 months until a few days ago when I decided it was time to get a real answer.

    Sean

    Comment | October 24, 2009
  • What are you using to create your diagrams? I would love to know!

    Comment | January 15, 2010
  • mattb

    I used the open office draw program and I founds some free scrap art icons to use. Glad you liked it. Take care.

    Comment | January 18, 2010
  • Manab

    Hi,
    I want to configure a webserver using the VM. Now I need to configure two separate network cards for both guest and host with separate Public IPs assigned to both the guest and host machines. Is it possible to configure in KVM (Qemu). If yes then how? Please help!

    Comment | February 17, 2010
  • mattb

    Manab,

    This article explains how to do that. Use the Virt-Manager tool to create your virtual machine. Then make the modifications based on the tutorial above.

    Basically you just have to manually add the ‘Spanning Tree’ capability to you interfaces to prevent Layer 2/Ethernet switching loops.

    -Matt

    Comment | February 17, 2010
  • mattb, thanks for sharing.
    Let me ask you something. Is it possible to create a bridge without a physical device?
    Here’s my situation, I have rented a server in a Datacenter. I do have eth0 conected to the internet, and want a couple of VMs working together like they were in a LAN, but eth1 is not connected to anything.
    I don’t want to bridge with eth0 for security reasons and that would cause routing problems when setting up a firewall between eth0 and the VMs.
    If I bridge with eth1, it simply doesn’t work because it’s not conected to a switch or anything… no link detected.

    Any suggestions on how to creat a local virtual network, where host and VMs can comunicate?

    Comment | March 16, 2010
  • mattb

    Alroger,

    I hope I understand the question correctly. To answer your question, yes. You can create two unbonded virtual interfaces. Although off the top of my head I couldn’t think of reason why one would do that. You would not be able to connect to a physical ethernet network. Meaning only those two VMs could talk to each other.

    The solution is to set your two VM’s to use eth1. Set eth1 to use spanning tree(if plugged into same switch as eth0)The tutorial above can help you get started on using spanning tree(if you need it) and how it applies to Ethernet networking.

    Read and study the OSI model.

    Comment | March 16, 2010
  • Whit

    Okay, two network cards plugged into the same switch … why? Is there no way to use the ability to assign multiple IPs to a single network card here?

    Comment | July 3, 2010
  • mattb

    You are correct, you can create virtual interfaces… but there are many reasons why I might want the option to use a separate physical device. Security and performance might be the top two issues, but what about ethernet specific applications? Like TDM over Ethernet?

    Asterisk PBX had TDM over Ethernet drivers for phone system hardware. I might want one interface(on my Asterisk virtual machine) to connect to an IP network and the other to be a gateway to bridged device.

    Comment | July 6, 2010
  • Do you have any suggestions about having a KVM host server with 4 physical NICs to be configured in such a way that individual VM instances can use exclusively one of the NICs ?

    Most (all?) of the KVM/Xen networking tutorials that I find consider only one physical NIC tied to a single bridge. A more realistic scenario should consider that most servers are delivered with more than 1 physical NIC. This would be of advantage to distribute network I/O load…

    Comment | November 13, 2012
  • mattb

    Easy follow my tutorial again but create more bridges. Assign a different bridge to each interface. Make sure STP is enabled!

    Comment | December 4, 2012

RSS feed for comments on this post. TrackBack URL

Leave a comment