Building Multi-Platform Docker Images with Travis CI and BuildKit

This is a lengthy note. If you don’t quite feel reading and only need the working example, go directly to the Travis CI build file.

The more I delve into the world of Raspberry Pi, the more I notice that “regular and boring” things on ARM are harder than I expected.

People build and distribute software exclusively for amd64. You read another “Kubernetes something” tutorial, that went viral on Twitter, and is fancy to try it out. Still, all helm charts, or whatever the author prefered, use Docker images built exclusively for amd64.

Docker toolchain has added the support for building multi-platform images in 19.x. However, it’s available only under the “experimental” mode. The topic of building multi-platform Docker images yet feels underrepresented.

But first, what are multi-platform Docker images?

When a client, e.g. Docker client, tries to pull an image, it must negotiate the details about what exactly to pull with the registry. The registry provides a manifest that describes the digest of the requested image, the volumes the image consists of, the platform this image can run on, etc. Optionally, the registry can provide a manifests list, which, as the name suggests, is a list of several manifests bundled into one. With the manifests list in hands, the client can figure out the particular digest of the image it needs to pull.

So multi-platform Docker images are just several images, whose manifests are bundled into the manifests list.

Imagine we want to pull the image golang:1.13.6-alpine3.10. Docker client will get the manifests list from Dockerhub. This list includes digests of several images, each built for the particular platform. If we’re on Raspberry Pi, running the current Raspbian Linux, which is arm/v7, the client will pick the corresponding image’s digest. Alternatively, we could choose to pull the image arm32v7/golang:1.13.6-alpine3.10 instead, and we ended up with the same image with the digest d72fa60fb5b9. Of course, to use a single universal image name, i.e. golang, on every platform is way more convenient.

You can read more about manifests in Docker registry documentation.

Does it mean I need to build different Docker images, for each platform I want to support?

Well, yes. This is how, official images are built.

For every platform, the image is built and pushed to the registry under the name <platform>/<image>:<tag>, e.g. amd64/golang:1-alpine. And next, a manifests list, that combines all those platform-specific images, is built and pushed with the simple name <image>:<tag>.

Docker’s BuildKit provides a toolkit that, among other nice things, allows building multi-platform images on a single host. BuildKit is used inside Docker’ buildx project, that is part of the recent Docker version.

One can use buildx, but, for this post, I wanted to try out, what would it look like to use BuildKit directly. For profefe, the system for continuous profiling of Go services, I set up Travis CI, that builds a multi-platform Docker image and pushes them to Dockerhub.

profefe is written in Go. That simplifies things, because, thanks to Go compiler, I don’t have to think about how to compile code for different platforms. The same Dockerfile will work fine on every platform.

Here’s how “deploy” stage of the build job looks like (see travis.yml on profefe’s GitHub).

dist: bionic

language: go
go:
  - 1.x

jobs:
  include:
    - stage: deploy docker
      services: docker
      env:
        - PLATFORMS="linux/amd64,linux/arm64,linux/arm/v7"
      install:
        - docker container run --rm --privileged multiarch/qemu-user-static --reset -p yes
        - docker container run -d --rm --name buildkitd --privileged moby/buildkit:latest
        - sudo docker container cp buildkitd:/usr/bin/buildctl /usr/local/bin/
        - export BUILDKIT_HOST="docker-container://buildkitd"
      script: skip
      deploy:
        - provider: script
          script: |
            buildctl build \
              --progress=plain \
              --frontend=dockerfile.v0 \
              --local context=. --local dockerfile=. \
              --opt filename=contrib/docker/Dockerfile \
              --opt platform=$PLATFORMS \
              --opt build-arg:VERSION=\"master\" \
              --opt build-arg:GITSHA=\"$TRAVIS_COMMIT\" \
              --output type=image,\"name=profefe/profefe:git-master\",push=true
          on:
            repo: profefe/profefe
            branch: master
      before_deploy:
        - echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin
      after_failure:
        - buildctl debug workers ls
        - docker container logs buildkitd

It’s a lot happening here, but I’ll describe the most critical parts.

Let’s start with dist: bionic.

We run the builds under Ubuntu 18.04 (Bionic Beaver). To be able to build multi-platform images on a single amd64 host, BuildKit uses QEMU to emulate other platforms. That requires Linux kernel 4.8, so even Ubuntu 16.04 (Xenial Xerus) should work.

The top-level details on how the emulation works are very well described in https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

In short, we tell the component of the kernel (binfmt_misc) to use QEMU when the system executes a binaries built for a different platform. The following call in the “install” step is what’s doing that:

- docker container run --rm --privileged multiarch/qemu-user-static --reset -p yes

Under the hood, the container runs a shell script from QEMU project, that registers the emulator as an executor of binaries from the external platforms.

If you think, that running a docker container to do the manipulations with the host’s OS looks weird, well… I can’t agree more. Probably, a better approach would be to install qemu-user-static, which would do the proper setup. Unfortunately, the current package’s version for Ubuntu Bionic doesn’t do the registration as we need it. I.e. its post-install doesn’t add the "F" flag (“fix binaries”), which is crucial for our goal. Let’s just agree,that docker-run will do ok for the demonstrational purpose.

- docker container run -d --rm --name buildkitd --privileged moby/buildkit:latest
- sudo docker container cp buildkitd:/usr/bin/buildctl /usr/local/bin/
- export BUILDKIT_HOST="docker-container://buildkitd"

This is another “docker-run’ism”. We start BuildKit’s buildkitd daemon inside the container, attaching it to the Docker daemon that runs on the host (“privileged” mode). Next, we copy buildctl binary from the container to the host system and set BUILDKIT_HOST environment variable, so buildctl knew where its daemon runs.

Alternatively, we could install BuildKit from GitHub and run the daemon directly on the build host. YOLO.

before_deploy:
  - echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin

To be able to push the images to the registry, we need to log in providing Docker credentials to host’s Docker daemon. The credentials are set as Travis CI’s encrypted environment variables ([refer to Travis CI docs])](https://docs.travis-ci.com/user/environment-variables/)).

buildctl build \
  --progress=plain \
  --frontend=dockerfile.v0 \
  --local context=. --local dockerfile=. \
  --opt filename=contrib/docker/Dockerfile \
  --opt platform=$PLATFORMS \
  --opt build-arg:VERSION=\"master\" \
  --opt build-arg:GITSHA=\"$TRAVIS_COMMIT\" \
  --output type=image,\"name=profefe/profefe:git-master\",push=true

This is the black box where everything happens. Magically!

We run buildctl stating that it must use the specified Dockerfile; it must build the images for defined platforms (I specified linux/amd64,linux/arm64,linux/arm/v7), create a manifests list tagged as the desired image (profefe/profefe:<version>), and push all the images to the registry.

buildctl debug workers ls shows what platforms does BuildKit on this host support. I listed only those I’m currently intrested with.

And that’s all. This setup automatically builds and pushes multi-platform Docker images for profefe (https://hub.docker.com/p/profefe/profefe) on a commit to project’s “master” branch on GitHub.

As I hope you’ve seen, support for multi-platform is getting easier and things that were hard a year ago are only mildly annoying now :)

If you have any comments or suggestions, reach out to me on Twitter or discuss this note on r/docker Reddit.

Some more reading on the topic:

All topics

applearduinoarm64askmeawsberlinbookmarksbuildkitcgocoffeecontinuous profilingCOVID-19designdockerdynamodbe-paperenglishenumesp8266firefoxgithub actionsGogooglegraphqlhomelabIPv6k3skuberneteslinuxmacosmaterial designmDNSmusicndppdneondatabaseobjective-cpasskeyspostgresqlpprofprofeferandomraspberry pirusttravis civs codewaveshareµ-benchmarks