k3s with Ubuntu Server (arm64) on Raspberry Pi 4
As I’ve twitted recently, I’m updating one of my Raspberry Pis to Ubuntu Server 19.10 (arm64).
“One of Raspberry Pis”?
My home cluster is four Raspberry Pis 4 (2GB); all connected to my internet router through ethernet and powered with 60W 6 USB-ports charger. All Pis build a small Kubernetes cluster that runs with k3s.
All by one Pis run on Raspbian Buster Lite and this setup’s been working pretty well until I’ve found out, Aerospike, a database I required to run for a testing lab, only works on a 64-bit OS.
Luckily, Ubuntu Server has an arm64 version built for Raspberry Pi. Thus, my working plan is to switch one Pi to Ubuntu, compile and run a single-instance Aerospike server (and any other components, that require a 64-bit OS) on this Pi, and provide a Kubernetes service in front of the DB, so other components in the cluster could access it as if it was fully managed by Kubernetes.
The Setup
Setting up Ubuntu Server on a Pi was smooth. All I did was flushing the image with 19.10 OS to an SD card, as described in Ubuntu wiki. That is, the headless setup worked out of the box, and after I inserted the card into the PI and connected it to the router, I managed to SSH into the system:
$ ssh ubuntu@192.168.10.18
The default password for ubuntu user is ubuntu. The system asks to change the password on the first login.
The first thing to do after installing the system:
$ sudo apt-get update
$ sudo apt-get upgrade -y
Disable “message of the day” (motd) to speed SSH login.
For that I commented out the following lines in /etc/pam.d/login and /etc/pam.d/sshd:
#session optional pam_motd.so motd=/run/motd.dynamic
#session optional pam_motd.so noupdate
Reduce GPU memory split. I truly don’t know if that even makes sense, tbh; read about memory split on Raspberry PI config-txt wiki. I added the following to /boot/firmware/usercfg.txt:
gpu_mem=16
To run Kubernetes or Docker, the kernel needs some cgroup options. On Ubuntu Server, the configuration is in /boot/firmware/nobtcmd.txt (refer to cmdline=nobtcmd.txt in /boot/firmware/nobtcfg.txt). Add the following to the end of the file:
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
Reboot the Pi, re-login, and all is ready to install k3s-agent:
$ curl -sfL https://get.k3s.io | K3S_URL="https://<k3s-master-pi>:6443" K3S_TOKEN="<k3s-token>" sh -
After the agent installed and running, check the Pi was added to Kubernetes cluster:
pi@pi-1:~ $ sudo kubectl get node -o wide
NAME ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pi-1 master 30d v1.17.0+k3s.1 192.168.10.16 Raspbian GNU/Linux 10 (buster) 4.19.75-v7l+ containerd://1.3.0-k3s.5
pi-2 <none> 47h v1.17.0+k3s.1 192.168.10.14 Raspbian GNU/Linux 10 (buster) 4.19.75-v7l+ containerd://1.3.0-k3s.5
pi-3 <none> 47h v1.17.0+k3s.1 192.168.10.15 Raspbian GNU/Linux 10 (buster) 4.19.75-v7l+ containerd://1.3.0-k3s.5
pi-4 <none> 10h v1.17.0+k3s.1 192.168.10.18 Ubuntu 19.10 5.3.0-1014-raspi2 containerd://1.3.0-k3s.5
That is for today. The next steps are to figure out how to build Aerospike on arm64, but this is a story for another day.
Update (2020-01-15)
I’ve managed to build and run Aerospike server for arm64! See make-arm64v8 branch in my fork of aerospike-server and the gist with my systemd services and configs.
Update (2020-02-13)
A week ago I tried to install Ubuntu Server 18.04.3 on Pi 4 and didn’t even get to the login shell in the headless mode. Now Ubuntu Server 18.04.4 LTS is out, and it works exactly as described in this note:
$ kubectl get node -o wide
NAME ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pi-1 master 63d v1.17.2+k3s1 192.168.10.16 Raspbian GNU/Linux 10 (buster) 4.19.93-v7l+ containerd://1.3.3-k3s1
pi-2 <none> 35d v1.17.2+k3s1 192.168.10.14 Raspbian GNU/Linux 10 (buster) 4.19.93-v7l+ containerd://1.3.3-k3s1
pi-3 <none> 14m v1.17.2+k3s1 192.168.10.20 Ubuntu 18.04.4 LTS 5.3.0-1017-raspi2 containerd://1.3.3-k3s1
pi-4 <none> 4d23h v1.17.2+k3s1 192.168.10.19 Ubuntu 19.10 5.3.0-1017-raspi2 containerd://1.3.3-k3s1
Update (2020-06-01)
Ubuntu Server 20.04 is the recommended version of Ubuntu for Ri 4. It works perfectly fine:
$ kubectl get node -o wide
NAME ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
pi-1 master 172d v1.18.2+k3s1 192.168.10.41 Raspbian GNU/Linux 10 (buster) 4.19.118-v7l+ containerd://1.3.3-k3s2
pi-2 <none> 57d v1.17.4+k3s1 192.168.10.42 Raspbian GNU/Linux 10 (buster) 4.19.118-v7l+ containerd://1.3.3-k3s2
pi-3 <none> 38d v1.17.4+k3s1 192.168.10.43 Ubuntu 20.04 LTS 5.4.0-1011-raspi containerd://1.3.3-k3s2
pi-4 <none> 3d19h v1.18.2+k3s1 192.168.10.44 Ubuntu 20.04 LTS 5.4.0-1011-raspi containerd://1.3.3-k3s2
Also Raspberry Pi OS (ex-Raspbian) for arm64 is currently in beta.