Falling deep down a networking rabbit hole, one hop at a time

One of my all time favorites https://xkcd.com/979/

Getting started

First things first, let’s launch a couple of hosts. This is super easy to do using Vagrant. I created two VirtualBox instances using standard Ubuntu distributions via the following configuration. I should add that they are connected on a private network.

$ vagrant up
Bringing machine 'host1' up with 'virtualbox' provider...
Bringing machine 'host2' up with 'virtualbox' provider...
==> host1: Importing base box 'ubuntu/bionic64'...
...
$ vagrant ssh host1
vagrant@host1:~$ ip a
....
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:65:8e:21 brd ff:ff:ff:ff:ff:ff
inet 172.28.128.3/24 brd 172.28.128.255 scope global dynamic
$ vagrant ssh host2
vagrant@host2:~$ ping 172.28.128.3
PING 172.28.128.3 (172.28.128.3) 56(84) bytes of data.
64 bytes from 172.28.128.3: icmp_seq=1 ttl=64 time=0.711 ms
64 bytes from 172.28.128.3: icmp_seq=2 ttl=64 time=0.487 ms

Network Namespaces + MacVTAP

Here’s where we get to the fun stuff. I wanted to use network namespaces for this project. If you’re not familiar with network namespaces, they’re a mechanism to provide a layer of isolation for the network stack in Linux. In general, many people operate in the default namespace without ever thinking about it. If you’ve ever used Docker networks, there’s a good chance you’ve used multiple network namespaces without even knowing. Creating a namespace can be done in a single command:

root@host2:~# ip netns add winter-sparkles
root@host2:~# ip netns list
winter-sparkles
root@host2:~# ip netns exec winter-sparkles bash
root@host2:~# ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
root@host2:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
root@host2:~# exit
### back in the default network namespace
root@host2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:62:a9:91:21:9f brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 85922sec preferred_lft 85922sec
inet6 fe80::62:a9ff:fe91:219f/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:c1:70:10 brd ff:ff:ff:ff:ff:ff
inet 172.28.128.4/24 brd 172.28.128.255 scope global dynamic enp0s8
valid_lft 722sec preferred_lft 722sec
inet6 fe80::a00:27ff:fec1:7010/64 scope link
valid_lft forever preferred_lft forever
root@host2:~# ip link add link enp0s8 name enp0s8vtap type macvtap mode bridge
root@host2:~# ip a
...
4: enp0s8vtap@enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
link/ether 92:f9:94:a2:99:8c brd ff:ff:ff:ff:ff:ff
root@host2:~# ip link set enp0s8vtap netns winter-sparkles up root@host2:~# ip netns exec winter-sparkles bash
root@host2:~# ip link set dev lo up
root@host2:~# ip address add 172.28.128.104/24 dev enp0s8vtap
#### back on host1
root@host1:~# ping 172.28.128.104
  • connect two hosts to a router
  • create a new interface on one of those hosts
  • assign it an address
  • ping to it
  • jump for joy as happiness ensues
#### host1
root@host1:~# tcpdump -nl -i any icmp &
root@host1:~# ping 172.28.128.104
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
04:46:21.727877 IP host1 > 172.28.128.104: ICMP echo request, id 2108, seq 39, length 64
04:46:22.778410 IP host1 > 172.28.128.104: ICMP echo request, id 2108, seq 40, length 64
#### host2 inside default namespace
root@host2:~# tcpdump -nl -i any icmp
#### host2 inside winter-sparkles namespace
root@host2:~# tcpdump -nl -i any icmp

Looking for martians

Ok, bare with me here. If your reaction to this heading is to think I’ve gone insane jumping straight to the theory that aliens have abducted my packets, you may be where I was when I first learned about martian packets. When a host receives packets on an interface and the source for the packet is routed on a different interface, it logs it as a martian or silently drops the packet. By default logging for martians is off in my machines, so I turned it on using sysctl:

#### host2 in both default & winter-sparkles
root@host2:~# sysctl -a | grep martians
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.default.log_martians = 0
root@host2:~# sysctl net.ipv4.conf.all.log_martians=1
root@host2:~# sysctl net.ipv4.conf.default.log_martians=1
root@host2:~# tail -f /var/log/kern.log | grep martian

Address Resolution Protocol

Running tcpdump with a host filter revealed some interesting details. I could see the Address Resolution Protocol (ARP) requests from host1 and the responses as well. Looking at the ARP table, it’s clear host1 was getting some traffic to the MacVTap interface.

#### host1
root@host1:~# tcpdump -nl -i any host 172.28.128.104 &
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
root@host1:~# ping 172.28.128.104
05:52:22.713348 ARP, Reply 172.28.128.104 is-at 66:20:3e:d8:81:b6, length 46

Finally

After getting some more sleep and coming back at it with fresh eyes and coffee, I thought of something else to try. Something I read in the VirtualBox docs make me think of it:

#### in the default namespace
root@host2:~# ip link set enp0s8 down
root@host2:~# ip link set enp0s8 address 66:20:3e:d8:81:b6
root@host2:~# ip link set enp0s8 up
#### in the winter-sparkles namespace
root@host2:~# ip link set enp0s8vtap down
root@host2:~# ip link set enp0s8vtap address 08:00:27:ae:cd:94
root@host2:~# ip link set enp0s8vtap up
#### back on host1
root@host1:~# ping 172.28.128.104
PING 172.28.128.104 (172.28.128.104) 56(84) bytes of data.
64 bytes from 172.28.128.104: icmp_seq=1 ttl=64 time=0.593 ms
64 bytes from 172.28.128.104: icmp_seq=2 ttl=64 time=0.726 ms

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alex Boten

Alex Boten

Passionate about the environment and making the world a better place