Last time, we set up a three layer container test environment:

So "Laptop" hosts "KVM" which hosts "Container". This lets us reconfigure and reboot the container host (the KVM system) without screwing up our real host environment (the Laptop system).

We ended with a shell prompt inside a container. Now we're going to set up networking in the container, with different routing than the KVM system so the Container system and KVM system have different views of the outside world.

LXC supports several different virtual network types, listed in the lxc.conf man page: veth uses Linux's ethernet bridging support, vlan sets up a virtual interface selects packets by IP address, and macvlan sets up a virtual interface that selects packets by mac address, that routes packets at the IP level, and veth joins interfaces together using Linux's ethernet bridging support (and the ebtables subsystem).

The other two networking options LXC supports are "empty" (just the loopback interface), and "phys" to move one of the host's ethernet interfaces into the container (removing it from the host system).

We're going to add a second ethernet interface to the KVM system, and use the "phys" option to move it into the container.

Step 1: Add a TAP interface to the Laptop.

The TUN/TAP subsystem creates a virtual ethernet interface attached to a process. (A TUN interface allows a userspace program to read/write IP packets, and a TAP interface works with ethernet frames instead.) For details, see the kernel TUN/TAP documentation.

We're going to attach a TAP interface to KVM, to add a second ethernet interface to the KVM system. Doing so requires root access on the laptop, but we can use the "tunctl" program (from the "uml-utilities" package) to create a new TUN/TAP interface and then hand it over to a non-root user (so we don't have to run KVM as root).

Save this as a shell script, you'll need to run it as root every time you reboot the host:

#!/bin/bash

if [ `id -u` -ne 0 ]
then
  echo "Needs root"
  exit 1
fi

# Replace "landley" with your username
tunctl -u landley -t kvm0
ifconfig kvm0 172.23.255.1 netmask 255.255.255.0
echo 1 > /proc/sys/net/ipv4/ip_forward
# replace eth0 with your outgoing interface (wlan0, usb0, etc).
# You can have more than one postrouting line if you regularly use different
# outgoing interfaces to access the net.
iptables -t nat -F POSTROUTING
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

The above commands last until the next time you reboot your Laptop system, at which point you'll have to re-run them. It's safe to re-run the script without rebooting.

The above commands create a tap interface "kvm0" on the laptop host and associate the address 172.23.255.1 with it. (Like 10.0.0.0/8 and 192.168.0.0/16, RFC 1918 also reserves 172.16.0.0/12 for private use.) It also makes kvm0 accessable to a specific regular user, and tells the Laptop to route packets between interfaces. The iptables rules allow it to access the rest of the internet as a masqueraded connection.

If you want to remove the tun/tap interface from the host (without rebooting), the command is:

tunctl -d kvm0

Step 2: Launch KVM with two ethernet interfaces.

We need to reboot our KVM system, still using the kernel and root filesystem we built last time but this time specifing two ethernet interfaces. The first is still eth0 masqueraded through a virtual 10.0.2.x LAN (for use by the KVM host), and the other's a TAP device connected directly to the host (for use by the container).

To do this, we need a more complicated KVM command line:


kvm -m 1024 -no-reboot -hda ~/squeeze.ext3 \
  -kernel arch/x86/boot/bzImage -append "root=/dev/hda rw panic=1" \
  -netdev user,id=lan0,hostfwd=tcp:127.0.0.1:9876-:22 \
  -device e1000,netdev=lan0 \
  -netdev tap,id=lan1,ifname=kvm0,script=no,downscript=no \
  -device e1000,netdev=lan1

This uses a newer syntax to add the devices, which allows us to specify which virtual network card plugs into which virtual network by supplying matching "id=" mechanisms.

The first virtual network (-netdev id=lan0) is a standard "user" virtual lan: it provides a masquerading gateway, dhcp, and a nameserver all in the 10.0.2.x address range. We plug the first e1000 card into that, which becomes eth0 in the KVM linux system.

The second virtual network (-netdev id=lan0) is a point to point connection to the TUN/TAP device we created on the host Laptop. We plug the second e1000 card into that, which becomes eth1 in the KVM linux system.

The hostfwd= argument forwards port 9876 of the laptop's loopback to port 22 on the kvm eth0 interface, giving us a way to ssh into the emulated system. (Since the first address the virtual dhcp server hands out defaults to 10.0.2.15, so we could explicitly say hostfwd=tcp:127.0.0.1:9876-10.0.2.15:22 if we wanted to, but if we only add one interface we can leave it out and it'll use the default.)

The script= arguments tell kvm not to fiddle with the host TUN/TAP interface, we set it up ourselves and it should leave it alone.

Step 3: Set up a new container in the KVM system.

To add a network interface to the container, we need a new configuration file in the format described by the "lxc.conf" man page. We're going to move a physical interface (eth1) from the host into the container. This will remove it from the host's namespace, and make it appear only in the container.

In the kvm system, go to the directory containing the static "busybox" binary and as root run:

cat > busybox.conf << EOF
lxc.utsname = busybox
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth1
#lxc.network.name = eth0
EOF

PATH=$(pwd):$PATH lxc-create -f busybox.conf -t busybox -n busybox
lxc-start -n busybox

The reason the last line of busybox.conf is commented out is to work around another bug: if the container's interface has the same name as the host interface, the two bleed together. So the host's eth1 interface will still be called "eth1" in the container, even though there's no eth0 there.

Leave that running and SSH into the KVM system again, get a shell prompt in the container and configure the container's new network interface:

lxc-console -n busybox

ifconfig eth1 172.23.255.2 netmask 255.255.255.0
route add default gw 172.23.255.1

Step 4: Fun with routing.

Now let's show that the container can access things the KVM can't. On the Laptop system, set up an alias of the loopback interface with the same IP address assigned to the KVM's eth0 (10.0.2.15). Then download the busybox binary to the Laptop and run busybox netcat in server mode so it prints "hello world" when you connect to port 12345.

sudo ifconfig lo:1 10.0.2.15 netmask 255.255.255.0
wget http://busybox.net/downloads/binaries/1.18.0/busybox-i686 -o busybox
chmod +x busybox
./busybox nc -p 12345 -lle echo hello world

Now from the container, try to connect to it with netcat:

nc 10.0.2.15 12345

It should print "hello world", meaning you connected to the laptop's lo:1 interface rather than the KVM's eth0. If you try the same command from the KVM system (./busybox nc 10.0.2.15 12345), it won't connect.