k3s with Ubuntu Server (arm64) on Raspberry Pi 4
As I’ve twitted recently, I’m updating one of my Raspberry Pis to Ubuntu Server 19.10 (arm64).
“One of Raspberry Pis”?
My home cluster is four Raspberry Pis 4 (2GB); all connected to my internet router through ethernet and powered with 60W 6 USB-ports charger. All Pis build a small Kubernetes cluster that runs with k3s.
All by one Pis run on Raspbian Buster Lite and this setup’s been working pretty well until I’ve found out, Aerospike, a database I required to run for a testing lab, only works on a 64-bit OS.
Luckily, Ubuntu Server has an arm64 version built for Raspberry Pi. Thus, my working plan is to switch one Pi to Ubuntu, compile and run a single-instance Aerospike server (and any other components, that require a 64-bit OS) on this Pi, and provide a Kubernetes service in front of the DB, so other components in the cluster could access it as if it was fully managed by Kubernetes.
Setting up Ubuntu Server on a Pi was smooth. All I did was flushing the image with 19.10 OS to an SD card, as described in Ubuntu wiki. That is, the headless setup worked out of the box, and after I inserted the card into the PI and connected it to the router, I managed to SSH into the system:
$ ssh firstname.lastname@example.org
The default password for
ubuntu user is
ubuntu. The system asks to change the password on the first login.
The first thing to do after installing the system:
$ sudo apt-get update $ sudo apt-get upgrade -y
Disable “message of the day” (
motd) to speed SSH login.
For that I commented out the following lines in
#session optional pam_motd.so motd=/run/motd.dynamic #session optional pam_motd.so noupdate
Reduce GPU memory split. I truly don’t know if that even makes sense, tbh; read about memory split on Raspberry PI config-txt wiki. I added the following to
To run Kubernetes or Docker, the kernel needs some cgroup options. On Ubuntu Server, the configuration is in
/boot/firmware/nobtcmd.txt (refer to
/boot/firmware/nobtcfg.txt). Add the following to the end of the file:
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
Reboot the Pi, re-login, and all is ready to install k3s-agent:
$ curl -sfL https://get.k3s.io | K3S_URL="https://<k3s-master-pi>:6443" K3S_TOKEN="<k3s-token>" sh -
After the agent installed and running, check the Pi was added to Kubernetes cluster:
pi@pi-1:~ $ sudo kubectl get node -o wide NAME ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME pi-1 master 30d v1.17.0+k3s.1 192.168.10.16 Raspbian GNU/Linux 10 (buster) 4.19.75-v7l+ containerd://1.3.0-k3s.5 pi-2 <none> 47h v1.17.0+k3s.1 192.168.10.14 Raspbian GNU/Linux 10 (buster) 4.19.75-v7l+ containerd://1.3.0-k3s.5 pi-3 <none> 47h v1.17.0+k3s.1 192.168.10.15 Raspbian GNU/Linux 10 (buster) 4.19.75-v7l+ containerd://1.3.0-k3s.5 pi-4 <none> 10h v1.17.0+k3s.1 192.168.10.18 Ubuntu 19.10 5.3.0-1014-raspi2 containerd://1.3.0-k3s.5
That is for today. The next steps are to figure out how to build Aerospike on arm64, but this is a story for another day.
I’ve managed to build and run Aerospike server for arm64! See make-arm64v8 branch in my fork of aerospike-server and the gist with my systemd services and configs.
A week ago I tried to install Ubuntu Server 18.04.3 on Pi 4 and didn’t even get to the login shell in the headless mode. Now Ubuntu Server 18.04.4 LTS is out, and it works exactly as described in this note:
$ kubectl get node -o wide NAME ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME pi-1 master 63d v1.17.2+k3s1 192.168.10.16 Raspbian GNU/Linux 10 (buster) 4.19.93-v7l+ containerd://1.3.3-k3s1 pi-2 <none> 35d v1.17.2+k3s1 192.168.10.14 Raspbian GNU/Linux 10 (buster) 4.19.93-v7l+ containerd://1.3.3-k3s1 pi-3 <none> 14m v1.17.2+k3s1 192.168.10.20 Ubuntu 18.04.4 LTS 5.3.0-1017-raspi2 containerd://1.3.3-k3s1 pi-4 <none> 4d23h v1.17.2+k3s1 192.168.10.19 Ubuntu 19.10 5.3.0-1017-raspi2 containerd://1.3.3-k3s1
Ubuntu Server 20.04 is the recommended version of Ubuntu for Ri 4. It works perfectly fine:
$ kubectl get node -o wide NAME ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME pi-1 master 172d v1.18.2+k3s1 192.168.10.41 Raspbian GNU/Linux 10 (buster) 4.19.118-v7l+ containerd://1.3.3-k3s2 pi-2 <none> 57d v1.17.4+k3s1 192.168.10.42 Raspbian GNU/Linux 10 (buster) 4.19.118-v7l+ containerd://1.3.3-k3s2 pi-3 <none> 38d v1.17.4+k3s1 192.168.10.43 Ubuntu 20.04 LTS 5.4.0-1011-raspi containerd://1.3.3-k3s2 pi-4 <none> 3d19h v1.18.2+k3s1 192.168.10.44 Ubuntu 20.04 LTS 5.4.0-1011-raspi containerd://1.3.3-k3s2
Also Raspberry Pi OS (ex-Raspbian) for arm64 is currently in beta.