Falling deep down a networking rabbit hole, one hop at a time

I ran into an interesting networking problem last week while configuring a multi-host Vagrant environment. I couldn’t find a lot of information about it anywhere so I thought, why not share it? Maybe it will save time for the next poor soul who tries to get VirtualBox virtual machines talking over MacVTap interfaces. Mostly I’m just hoping that I’m not alone out there trying to make this work.

One of my all time favorites https://xkcd.com/979/

Getting started

Once the hosts come up, I run a simple ping to make sure everything works. Looking good so far:

$ vagrant up
Bringing machine 'host1' up with 'virtualbox' provider...
Bringing machine 'host2' up with 'virtualbox' provider...
==> host1: Importing base box 'ubuntu/bionic64'...
...
$ vagrant ssh host1
vagrant@host1:~$ ip a
....
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:65:8e:21 brd ff:ff:ff:ff:ff:ff
inet 172.28.128.3/24 brd 172.28.128.255 scope global dynamic
$ vagrant ssh host2
vagrant@host2:~$ ping 172.28.128.3
PING 172.28.128.3 (172.28.128.3) 56(84) bytes of data.
64 bytes from 172.28.128.3: icmp_seq=1 ttl=64 time=0.711 ms
64 bytes from 172.28.128.3: icmp_seq=2 ttl=64 time=0.487 ms

Network Namespaces + MacVTAP

root@host2:~# ip netns add winter-sparkles
root@host2:~# ip netns list
winter-sparkles

You can then hop into your namespace and poke around:

root@host2:~# ip netns exec winter-sparkles bash
root@host2:~# ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
root@host2:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
root@host2:~# exit

MacVTap is a Linux device driver that allows us to create virtual interfaces with their own IP and physical addresses. With it, we’re able to create as many virtual interfaces as we need in order to provide each network namespace with its own interface. There are other mechanisms that can accomplish this (queue acronym soup), but MacVTap is what I was working with. Creating a MacVTap interface looks like:

### back in the default network namespace
root@host2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:62:a9:91:21:9f brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 85922sec preferred_lft 85922sec
inet6 fe80::62:a9ff:fe91:219f/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:c1:70:10 brd ff:ff:ff:ff:ff:ff
inet 172.28.128.4/24 brd 172.28.128.255 scope global dynamic enp0s8
valid_lft 722sec preferred_lft 722sec
inet6 fe80::a00:27ff:fec1:7010/64 scope link
valid_lft forever preferred_lft forever
root@host2:~# ip link add link enp0s8 name enp0s8vtap type macvtap mode bridge
root@host2:~# ip a
...
4: enp0s8vtap@enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
link/ether 92:f9:94:a2:99:8c brd ff:ff:ff:ff:ff:ff

I’ll then take the interface and add it to the namespace I created earlier. To make the interface useful, we’ll need to give it an IP address:

root@host2:~# ip link set enp0s8vtap netns winter-sparkles up root@host2:~# ip netns exec winter-sparkles bash
root@host2:~# ip link set dev lo up
root@host2:~# ip address add 172.28.128.104/24 dev enp0s8vtap
#### back on host1
root@host1:~# ping 172.28.128.104

And that’s when the problems started. On a physical network, the following would work:

  • connect two hosts to a router
  • create a new interface on one of those hosts
  • assign it an address
  • ping to it
  • jump for joy as happiness ensues

Here though, complete silence. Time to crack open tcpdump:

#### host1
root@host1:~# tcpdump -nl -i any icmp &
root@host1:~# ping 172.28.128.104
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
04:46:21.727877 IP host1 > 172.28.128.104: ICMP echo request, id 2108, seq 39, length 64
04:46:22.778410 IP host1 > 172.28.128.104: ICMP echo request, id 2108, seq 40, length 64
#### host2 inside default namespace
root@host2:~# tcpdump -nl -i any icmp
#### host2 inside winter-sparkles namespace
root@host2:~# tcpdump -nl -i any icmp

Nothing. The pings never make it to the second host, either inside or outside of the namespace. Let’s widen the search and learn.

Looking for martians

NOTE: sysctl will only apply within their namespace, you’ll need to apply them in both namespaces

#### host2 in both default & winter-sparkles
root@host2:~# sysctl -a | grep martians
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.default.log_martians = 0
root@host2:~# sysctl net.ipv4.conf.all.log_martians=1
root@host2:~# sysctl net.ipv4.conf.default.log_martians=1
root@host2:~# tail -f /var/log/kern.log | grep martian

Nope, not that either, exhausted another theory. Onto the next hop.

Address Resolution Protocol

#### host1
root@host1:~# tcpdump -nl -i any host 172.28.128.104 &
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
root@host1:~# ping 172.28.128.104
05:52:22.713348 ARP, Reply 172.28.128.104 is-at 66:20:3e:d8:81:b6, length 46

Looking through the documentation for internal network configuration in VirtualBox didn’t give me a lot to go on. I started poking around the user interface. I knew I was really grasping at straws but I was running out of options I could think of. I found the “Promiscuous Mode” option and set it from “Deny” to “Allow”, still no changes.

Finally

Once there is more than one active virtual network card with the same internal network ID, the Oracle VM VirtualBox support driver will automatically wire the cards and act as a network switch

Reading that description, I wondered if spoofing the Media Access Control (MAC) address of the device would allow my virtual interface to connect to the network. Spoofing means to act as something else, in my case, the MacVTap interface would pretend to be the original interface VirtualBox created.

#### in the default namespace
root@host2:~# ip link set enp0s8 down
root@host2:~# ip link set enp0s8 address 66:20:3e:d8:81:b6
root@host2:~# ip link set enp0s8 up
#### in the winter-sparkles namespace
root@host2:~# ip link set enp0s8vtap down
root@host2:~# ip link set enp0s8vtap address 08:00:27:ae:cd:94
root@host2:~# ip link set enp0s8vtap up
#### back on host1
root@host1:~# ping 172.28.128.104
PING 172.28.128.104 (172.28.128.104) 56(84) bytes of data.
64 bytes from 172.28.128.104: icmp_seq=1 ttl=64 time=0.593 ms
64 bytes from 172.28.128.104: icmp_seq=2 ttl=64 time=0.726 ms

At long last, there it is. This means that as far as I understand, I will not be able to use MacVTap interfaces if I want to use internal networks, as it appears there is some MAC filtering happening here. Days have gone by, but dammit it feels amazing to have gotten to the bottom of the missing packets. Of course, by this point I have no idea what I was originally trying to do. Would love to hear from others that may have gotten further along this route!

Passionate about the environment and making the world a better place