k3s with Ubuntu Server (arm64) on Raspberry Pi 4

As I’ve twitted recently, I’m updating one of my Raspberry Pis to Ubuntu Server 19.10 (arm64).

“One of Raspberry Pis”?

My home cluster is four Raspberry Pis 4 (2GB); all connected to my internet router through ethernet and powered with 60W 6 USB-ports charger. All Pis build a small kubernetes cluster that runs with k3s.

All by one Pis run on Raspbian Buster Lite and this setup’s been working pretty well until I’ve found out, Aerospike, a database I required to run for a testing lab, only works on a 64-bit OS.

Luckily, Ubuntu Server has an arm64 version built for Raspberry Pi. Thus, my working plan is to switch one Pi to Ubuntu, compile and run a single-instance Aerospike server (and any other components, that require a 64-bit OS) on this Pi, and provide a kubernetes service in front of the DB, so other components in the cluster could access it as if it was fully managed by kubernetes.

The Setup

Setting up Ubuntu Server on a Pi was smooth. All I did was flushing the image with 19.10 OS to an SD card, as described in Ubuntu wiki. That is, the headless setup worked out of the box, and after I inserted the card into the PI and connected it to the router, I managed to SSH into the system:

$ ssh ubuntu@192.168.10.18

The default password for ubuntu user is ubuntu. The system asks to change the password on the first login.

The first thing to do after installing the system:

$ sudo apt-get update
$ sudo apt-get upgrade -y

Disable “message of the day” (motd) to speed SSH login. For that I commented out the following lines in /etc/pam.d/login and /etc/pam.d/sshd:

#session    optional     pam_motd.so  motd=/run/motd.dynamic
#session    optional     pam_motd.so noupdate

Reduce GPU memory split. I truly don’t know if that even makes sense, tbh; read about memory split on Raspberry PI config-txt wiki. I added the following to /boot/firmware/usercfg.txt:

gpu_mem=16

To run Kubernetes or Docker, the kernel needs some cgroup options. On Ubuntu Server, the configuration is in /boot/firmware/nobtcmd.txt (refer to cmdline=nobtcmd.txt in /boot/firmware/nobtcfg.txt). Add the following to the end of the file:

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Reboot the Pi, re-login, and all is ready to install k3s-agent:

$ curl -sfL https://get.k3s.io  | K3S_URL="https://<k3s-master-pi>:6443" K3S_TOKEN="<k3s-token>" sh -

After the agent installed and running, check the Pi was added to kubernetes cluster:

pi@pi-1:~ $ sudo kubectl get node -o wide
NAME   STATUS   ROLES    AGE   VERSION         INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION      CONTAINER-RUNTIME
pi-1   Ready    master   30d   v1.17.0+k3s.1   192.168.10.16   <none>        Raspbian GNU/Linux 10 (buster)   4.19.75-v7l+        containerd://1.3.0-k3s.5
pi-2   Ready    <none>   47h   v1.17.0+k3s.1   192.168.10.14   <none>        Raspbian GNU/Linux 10 (buster)   4.19.75-v7l+        containerd://1.3.0-k3s.5
pi-3   Ready    <none>   47h   v1.17.0+k3s.1   192.168.10.15   <none>        Raspbian GNU/Linux 10 (buster)   4.19.75-v7l+        containerd://1.3.0-k3s.5
pi-4   Ready    <none>   10h   v1.17.0+k3s.1   192.168.10.18   <none>        Ubuntu 19.10                     5.3.0-1014-raspi2   containerd://1.3.0-k3s.5

That is for today. The next steps are to figure out how to build Aerospike on arm64, but this is a story for another day.

Update (2020-01-15)

I’ve managed to build and run Aerospike server for arm64! See make-arm64v8 branch in my fork of aerospike-server and the gist with my systemd services and configs.

Update (2020-02-13)

A week ago I tried to install Ubuntu Server 18.04.3 on Pi 4 and didn’t even get to the login shell in the headless mode. Now Ubuntu Server 18.04.4 LTS is out and it works exactly as described in this note:

$ kubectl get node -o wide
NAME   STATUS   ROLES    AGE     VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION      CONTAINER-RUNTIME
pi-1   Ready    master   63d     v1.17.2+k3s1   192.168.10.16   <none>        Raspbian GNU/Linux 10 (buster)   4.19.93-v7l+        containerd://1.3.3-k3s1
pi-2   Ready    <none>   35d     v1.17.2+k3s1   192.168.10.14   <none>        Raspbian GNU/Linux 10 (buster)   4.19.93-v7l+        containerd://1.3.3-k3s1
pi-3   Ready    <none>   14m     v1.17.2+k3s1   192.168.10.20   <none>        Ubuntu 18.04.4 LTS               5.3.0-1017-raspi2   containerd://1.3.3-k3s1
pi-4   Ready    <none>   4d23h   v1.17.2+k3s1   192.168.10.19   <none>        Ubuntu 19.10                     5.3.0-1017-raspi2   containerd://1.3.3-k3s1