Wireless-to-Ethernet island for homelab cluster: IPv6, NDP proxy and mDNS reflector
Initially, when I assembled a homelab cluster of Raspberry Pis, everything was directly connected to my Wi-Fi router with the Ethernet cables. This worked fine but this “stack of boards” behind the sofa in the centre of our small flat bugged me a bit.
Last year I decided to reorganise the cluster, turning it into a wireless-to-wired island, which I could relocate anywhere within the flat, without doing any special cable management, while staying cheap and avoid stacking the appartment with even more gadgets. After going through a number of trials and errors, the final setup looks as the following:
Different colours contour the connections between two logical subnets — more on that later. Here what we have on the schema (top to bottom):
- A cable modem + Wi-Fi router, which I received from the ISP.
The router is configured with a private IPv4 address 192.168.10.1 and a global unicast IPv6 address (GUA) prefix 2001:db8:abc:123::/64, which the ISP designates to us (of course, that’s not the real prefix, but I will use this one in all examples below).
Right from the beginning, it’s worth to mention, that the router (Sagemcom F@st) is super limited in what it allows configuring. E.g. according to the internet, my ISP — Vodafone Germany — provides /60 or /62 IPv6 prefix per a private customer. With that, I intended to divide my home network into two /64 subnets: one for the homelab and another one for the rest of Wi-Fi devices we use at home. It turned out the router sets a fixed /64, without any way to change it. Similarly, I can’t set up any custom routing or configure DNS servers, that the router’s built-in DHCP provides for the home network — the configuration is locked by the vendor 😑
pi-31(Raspberry Pi 3, 1GB) — the homelab cluster’s gateway.
pi-31 connects to the Wi-Fi thought its wlan0 interface, which I configured with a static IPv4 address 192.168.10.31 and a global IPv6 address 2001:db8:abc:123::YYY/64, it autoconfigures by itself, thanks to SLAAC.
The gateway manages a standalone local network and routes the traffic between the wireless and the local network. Its eth0 interface has a static IPv4 address 192.168.20.1 and a unique local IPv6 address (ULA) fd2f::1/64 — this is the LAN, where the cluster of Raspberry Pis lives.
pi-31 runs dnsmasq to provide management services to the cluster: DHCP, Router Advertisements (RA) and DNS.
pi-41..pi-44(4x Raspberry Pi 4, 2GB) — the cluster’s “worker” nodes.
Every pi-4x connects to the gateway thought a network switch. DHCP server on the gateway assigns a static IPv4 address 192.168.20.4x/24 and announces a ULA prefix fd2f::/64, that hosts use to autoconfigure their IPv6 address and the default route.
In order to provide the access from the cluster’s LAN to the internet, the gateway sets NAT (IP masquerading and IPv4 routing). There are a ton of articles on how to configure that for Raspberry Pi (e.g. “Setting up a Raspberry Pi as a routed wireless access point”) — I will skip this part.
However, there are several annoying things about this setup (and about NAT in particular):
I can’t connect to pi-4x hosts directly from my laptop, which lives on the main home network. That is, if I want to SSH to pi-41, I either have to use pi-31 as a jump host:
$ ssh -J pi@192.168.10.31 pi@192.168.20.41
Or, I need to define the static route on the laptop system’s routing table. On macOS I did that with the following command:
$ sudo route -n add 192.168.20.0/24 192.168.10.31
As I mentioned above, my router doesn’t let me configure static routes, so I had to do it on all device from where I needed to access pi-4x hosts.
Second, mDNS doesn’t work, even though every RPi-host runs avahi-daemon. That is, in the SSH example above, I had to use host’s IP address instead of simply ssh pi-41.local.
Last, IPv6 internet doesn’t work from the cluster. I could set NATv6 on the gateway, in the same manner I set it for IPv4 addresses, but that feels boring :)
Fixing the lack of IPv6 internet inside the cluster
The Wi-Fi router gives us /64 GUA prefix. I divided it into a smaller subnet 2001:db8:abc:123:40::/76 and gave the IPs from this subnet to pi-4x hosts. SLAAC isn’t the option for subnets smaller than /64 so I had to assign a global IPv6 address from the chosen prefix to each worker host statically.
Observe that each host’s eth0 has three IPv6 addresses (I’ve omitted the output for pi-43 and pi-44 for brevity):
pi@pi-41: $ ip -6 addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 fd2f:4a6d:83e5:1:deb0:7ee0:4d1:b11a/64 2️⃣ scope global dynamic {··}
inet6 2001:db8:abc:123:41::1/80 3️⃣ scope global noprefixroute {··}
inet6 fe80::f388:8cca:fb1d:c8ec/64 1️⃣ scope link {··}
pi@pi-42: ip -6 addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 fd2f:4a6d:83e5:1:840:35d7:76d:9b43/64 2️⃣ scope global dynamic {··}
inet6 2001:db8:abc:123:42::1/80 3️⃣ scope global noprefixroute {··}
inet6 fe80::3152:8c62:a0e6:c206/64 1️⃣ scope link {··}
- a link-local address
fe80::/64 - a ULA address
fd2f::/64, autoconfigured with SLAAC after the prefix announced by the gateway - a global address, I configured manually
2001:db8:abc:123:4X::1/80, whereXis the number, matched with thexinpi-4x(I’ve set the address with/80prefix, but that isn’t important here).
When a pi-4x host autoconfigures its ULA address it also sets the default IPv6 route with the link-local address of the gateway:
pi@pi-31: $ ip -6 addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
···
inet6 fe80::b139:44c0:f524:eed5/64 1️⃣ scope link
pi@pi-41: $ ip -6 route show default
default via fe80::b139:44c0:f524:eed5 2️⃣ dev eth0 proto ra metric 202 mtu 1500 pref medium
Above, pi-31’s link-local address fe80::/64 1️⃣ is set as the default route 2️⃣ on pi-41 — it’s the same on all other pi-4x hosts. This allows routing the IPv6 traffic, whose destination address doesn’t match any other routing rule, though pi-31 gateway.
However, if a pi-4x host tried to ping an external IPv6 address, e.g. ping -6 ya.ru, none of the packets would get back to it. Even though the host has a global IPv6 address, the Wi-Fi router, who receives the response to a ping from the WAN, doesn’t know where to route the packet. When the router performs “neighbour discovery” (ND), it sends a multicast packet to all local devices, asking for a MAC address of the owner of the IPv6 address of the packet’s destination. Since the pi-4x hosts aren’t directly connected to the Wi-Fi router, they won’t receive the multicast and won’t be able to response to the ND (this explanation of Neighbour Discovery in IPv6 was very helpful, when I was trying to wrap my head around the topic).
To fix this, I configured NDP proxying on pi-31 gateway, with the help of ndppd
ndppdis a daemon that proxies certain IPv6 NDP messages between two or more interfaces.
The config in /etc/ndppd.conf makes pi-31, who has the direct connection to the Wi-Fi router through wlan0, to answer to neighbour solicitation requests for all addresses in 2001:db8:abc:123:40::/76 subnet:
pi@pi-31: $ cat /etc/ndppd.conf
proxy wlan0 {
autowire yes
rule 2001:db8:abc:123:40::/76 {
iface eth0
}
}
I also set a static route on pi-31 so it routed all traffic to 2001:db8:abc:123:40::/76 through eth0, forwarding the traffic to the corresponding pi-4x hosts:
pi@pi-31:~ $ ip -6 route show
2001:db8:abc:123:40::/76 dev eth0 metric 100 pref medium
2001:db8:abc:123::/64 dev wlan0 proto ra metric 303 mtu 1500 pref medium
···
I’ve put the route into /etc/dhcpcd.exit-hook on pi-31 so the configuration persisted between the restarts.
With NDP proxying and the route in place, pi-4x hosts can finally ping the IPv6 internet:
pi@pi-42: $ ping -6 ya.ru
PING ya.ru(ya.ru (2a02:6b8::2:242)) 56 data bytes
64 bytes from ya.ru (2a02:6b8::2:242): icmp_seq=1 ttl=52 time=54.1 ms
64 bytes from ya.ru (2a02:6b8::2:242): icmp_seq=2 ttl=52 time=57.4 ms
As an additional side effect, because the IPv6 traffic from the Wi-Fi router is properly routed to the homelab cluster, I no longer have to use jump host when SSH to a pi-4x host — I can use host’s global IPv6 address instead:
$ ssh pi@2001:db8:abc:123:42::1
Last login: Sun Apr 25 17:19:33 2021 from 2001:db8:abc:123:99b9:6677:751b:dc75
pi@pi-42:~ $
Fixing mDNS
avahi-daemon can reflect the incoming mDNS requests to all local network interfaces on the host out of the box. All I have to do was to specify enable-reflector=yes in /etc/avahi/avahi-daemon.conf on pi-31 gateway. Refer to man avahi-daemon.conf for more details about the available knobs.
Now I can access any pi-4x host from the main home network with mDNS, just as if we were on the same LAN:
$ ping6 pi-41.local
PING6(56=40+8+8 bytes) 2001:db8:abc:123:99b9:6677:751b:dc75 --> 2001:db8:abc:123:41::1
16 bytes from 2001:db8:abc:123:41::1, icmp_seq=0 hlim=63 time=5.536 ms
16 bytes from 2001:db8:abc:123:41::1, icmp_seq=1 hlim=63 time=4.003 ms
$ ssh pi@pi-41.local
Last login: Sun Apr 25 17:48:54 2021 from 2001:db8:abc:123:99b9:6677:751b:dc75
pi@pi-41:~ $
Of course, this works only via IPv6. Setting up an ARP proxying on pi-31 gateway will probably fix it for IPv4 too, but that could be a story for another day. I’m already very happy with the results.
Solving the outlined problems is what made the whole reorganisation process fun. I had to delve into a bunch of new things, although I’m sure there are ways I could do it better. Discuss this note on Twitter and Hacker News.