Falling deep down a networking rabbit hole, one hop at a time

6 min readFeb 1, 2019

I ran into an interesting networking problem last week while configuring a multi-host Vagrant environment. I couldn’t find a lot of information about it anywhere so I thought, why not share it? Maybe it will save time for the next poor soul who tries to get VirtualBox virtual machines talking over MacVTap interfaces. Mostly I’m just hoping that I’m not alone out there trying to make this work.

One of my all time favorites https://xkcd.com/979/

Getting started

First things first, let’s launch a couple of hosts. This is super easy to do using Vagrant. I created two VirtualBox instances using standard Ubuntu distributions via the following configuration. I should add that they are connected on a private network.

Once the hosts come up, I run a simple ping to make sure everything works. Looking good so far:

$ vagrant up
Bringing machine 'host1' up with 'virtualbox' provider...
Bringing machine 'host2' up with 'virtualbox' provider...
==> host1: Importing base box 'ubuntu/bionic64'...
...$ vagrant ssh host1
vagrant@host1:~$ ip a
....
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:65:8e:21 brd ff:ff:ff:ff:ff:ff
    inet 172.28.128.3/24 brd 172.28.128.255 scope global dynamic$ vagrant ssh host2
vagrant@host2:~$ ping 172.28.128.3
PING 172.28.128.3 (172.28.128.3) 56(84) bytes of data.
64 bytes from 172.28.128.3: icmp_seq=1 ttl=64 time=0.711 ms
64 bytes from 172.28.128.3: icmp_seq=2 ttl=64 time=0.487 ms

Network Namespaces + MacVTAP

Here’s where we get to the fun stuff. I wanted to use network namespaces for this project. If you’re not familiar with network namespaces, they’re a mechanism to provide a layer of isolation for the network stack in Linux. In general, many people operate in the default namespace without ever thinking about it. If you’ve ever used Docker networks, there’s a good chance you’ve used multiple network namespaces without even knowing. Creating a namespace can be done in a single command:

root@host2:~# ip netns add winter-sparkles
root@host2:~# ip netns list
winter-sparkles

You can then hop into your namespace and poke around:

root@host2:~# ip netns exec winter-sparkles bash
root@host2:~# ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
root@host2:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
root@host2:~# exit

MacVTap is a Linux device driver that allows us to create virtual interfaces with their own IP and physical addresses. With it, we’re able to create as many virtual interfaces as we need in order to provide each network namespace with its own interface. There are other mechanisms that can accomplish this (queue acronym soup), but MacVTap is what I was working with. Creating a MacVTap interface looks like:

### back in the default network namespace
root@host2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:62:a9:91:21:9f brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 85922sec preferred_lft 85922sec
    inet6 fe80::62:a9ff:fe91:219f/64 scope link
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:c1:70:10 brd ff:ff:ff:ff:ff:ff
    inet 172.28.128.4/24 brd 172.28.128.255 scope global dynamic enp0s8
       valid_lft 722sec preferred_lft 722sec
    inet6 fe80::a00:27ff:fec1:7010/64 scope link
       valid_lft forever preferred_lft forever
root@host2:~# ip link add link enp0s8 name enp0s8vtap type macvtap mode bridge
root@host2:~# ip a
...
4: enp0s8vtap@enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
    link/ether 92:f9:94:a2:99:8c brd ff:ff:ff:ff:ff:ff

I’ll then take the interface and add it to the namespace I created earlier. To make the interface useful, we’ll need to give it an IP address:

root@host2:~# ip link set enp0s8vtap netns winter-sparkles up root@host2:~# ip netns exec winter-sparkles bash
root@host2:~# ip link set dev lo up
root@host2:~# ip address add 172.28.128.104/24 dev enp0s8vtap #### back on host1
root@host1:~# ping 172.28.128.104

And that’s when the problems started. On a physical network, the following would work:

connect two hosts to a router
create a new interface on one of those hosts
assign it an address
ping to it
jump for joy as happiness ensues

Here though, complete silence. Time to crack open tcpdump:

#### host1
root@host1:~# tcpdump -nl -i any icmp &
root@host1:~# ping 172.28.128.104
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
04:46:21.727877 IP host1 > 172.28.128.104: ICMP echo request, id 2108, seq 39, length 64
04:46:22.778410 IP host1 > 172.28.128.104: ICMP echo request, id 2108, seq 40, length 64#### host2 inside default namespace
root@host2:~# tcpdump -nl -i any icmp#### host2 inside winter-sparkles namespace
root@host2:~# tcpdump -nl -i any icmp

Nothing. The pings never make it to the second host, either inside or outside of the namespace. Let’s widen the search and learn.

Looking for martians

Ok, bare with me here. If your reaction to this heading is to think I’ve gone insane jumping straight to the theory that aliens have abducted my packets, you may be where I was when I first learned about martian packets. When a host receives packets on an interface and the source for the packet is routed on a different interface, it logs it as a martian or silently drops the packet. By default logging for martians is off in my machines, so I turned it on using sysctl:

NOTE: sysctl will only apply within their namespace, you’ll need to apply them in both namespaces

#### host2 in both default & winter-sparkles
root@host2:~# sysctl -a | grep martians
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.default.log_martians = 0
root@host2:~# sysctl net.ipv4.conf.all.log_martians=1
root@host2:~# sysctl net.ipv4.conf.default.log_martians=1
root@host2:~# tail -f /var/log/kern.log | grep martian

Nope, not that either, exhausted another theory. Onto the next hop.

Address Resolution Protocol

Running tcpdump with a host filter revealed some interesting details. I could see the Address Resolution Protocol (ARP) requests from host1 and the responses as well. Looking at the ARP table, it’s clear host1 was getting some traffic to the MacVTap interface.

#### host1
root@host1:~# tcpdump -nl -i any host 172.28.128.104 &
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
root@host1:~# ping 172.28.128.104
05:52:22.713348 ARP, Reply 172.28.128.104 is-at 66:20:3e:d8:81:b6, length 46

Looking through the documentation for internal network configuration in VirtualBox didn’t give me a lot to go on. I started poking around the user interface. I knew I was really grasping at straws but I was running out of options I could think of. I found the “Promiscuous Mode” option and set it from “Deny” to “Allow”, still no changes.

Finally

After getting some more sleep and coming back at it with fresh eyes and coffee, I thought of something else to try. Something I read in the VirtualBox docs make me think of it:

Once there is more than one active virtual network card with the same internal network ID, the Oracle VM VirtualBox support driver will automatically wire the cards and act as a network switch

Reading that description, I wondered if spoofing the Media Access Control (MAC) address of the device would allow my virtual interface to connect to the network. Spoofing means to act as something else, in my case, the MacVTap interface would pretend to be the original interface VirtualBox created.

#### in the default namespace
root@host2:~# ip link set enp0s8 down
root@host2:~# ip link set enp0s8 address 66:20:3e:d8:81:b6
root@host2:~# ip link set enp0s8 up#### in the winter-sparkles namespace
root@host2:~# ip link set enp0s8vtap down
root@host2:~# ip link set enp0s8vtap address 08:00:27:ae:cd:94
root@host2:~# ip link set enp0s8vtap up#### back on host1
root@host1:~# ping 172.28.128.104
PING 172.28.128.104 (172.28.128.104) 56(84) bytes of data.
64 bytes from 172.28.128.104: icmp_seq=1 ttl=64 time=0.593 ms
64 bytes from 172.28.128.104: icmp_seq=2 ttl=64 time=0.726 ms

At long last, there it is. This means that as far as I understand, I will not be able to use MacVTap interfaces if I want to use internal networks, as it appears there is some MAC filtering happening here. Days have gone by, but dammit it feels amazing to have gotten to the bottom of the missing packets. Of course, by this point I have no idea what I was originally trying to do. Would love to hear from others that may have gotten further along this route!