Multi-Arch Registry and Images

It has been a while since multi-architecture support has been developed and brought upstream. Multi-architecture support allows to put several images behind a common identifier in a registry. With that, a consumer of the image won't have to think about pulling the right binary flavor of the image, it's done right automatically for him.

A couple of things play together for this magic:

1. A manifest list defines the content of a (composite) image. In this manifest list, images are specified and can be flagged with OS type, architecture, and optional features.
2. A manifest tool uses the manifest list and assembles the composite image in a registry. Today's tool is a stop gap until this will finally be picked up by some docker tool (docker "push" command?), but some details need to be fleshed out, like signing or conventions on the use of all the manifest list flags.
3. A docker registry, version 2.3 or higher. Docker Hub also appears to work, while Docker Trusted Registry is not there today. The registry stores the metadata and manifest to serve the composite image.
4. The docker engine will pull the right image layers from the registry according to its OS type and architecture

Let's try this out. We'll use a private registry, version 2.3 or later. The previous post describes how to get that. In this example, the registry is at hostname s8345002.
We assume we have pushed a z image ("s8345002:5000/demo-s390x:v0.2") and an x86 image ("s8345002:5000/demo-amd64:v0.2") into the registry.

Let's go for the manifest tool. I'm using an Ubuntu system with golang installed. You could run a container off the s390x/ubuntu image and and install git and golang to get to a similar build environment. Alternatively, a gccgo environment is also good. Check out previous posts (here and here) for options to get a go environment.

root@s8345002:~# export GOPATH=/go
root@s8345002:~# mkdir -p /go/src/github.com/estesp
root@s8345002:~# cd /go/src/github.com/estesp
root@s8345002:/go/src/github.com/estesp# git clone https://github.com/estesp/manifest-tool
Cloning into 'manifest-tool'...
remote: Counting objects: 2290, done.
remote: Total 2290 (delta 0), reused 0 (delta 0), pack-reused 2290
Receiving objects: 100% (2290/2290), 1.33 MiB | 865.00 KiB/s, done.
Resolving deltas: 100% (845/845), done.
Checking connectivity... done.
root@s8345002:/go/src/github.com/estesp# cd manifest-tool/
root@s8345002:/go/src/github.com/estesp/manifest-tool# make binary
go build -o manifest github.com/estesp/manifest-tool
root@s8345002:/go/src/github.com/estesp/manifest-tool# file manifest
manifest: ELF 64-bit MSB shared object, IBM S/390, version 1 (SYSV), dynamically linked, interpreter /lib/ld64.so.1, for GNU/Linux 3.2.0, BuildID[sha1]=4558a22a1bd2855cf8cb9d909e6491dacef92b61, not stripped

Update 5/24: instead of make, use make binary meanwhile

You can use "make install" to put the binary into /usr/bin (warning to Ubuntu users: it will overwrite your existing manifest tool, which is a packaging tool, so maybe avoid this step), or just copy it someplace to use it as appropriate.

Usage of the manifest tool is simple; it accepts two commands, pushml and inspect. pushml will take a manifest list and create a composite image in a registry. In our case, the manifest list is in a file demo_v0.2.yml and looks like this:

image: s8345002:5000/demo:v0.2
manifests:
  -
    image: s8345002:5000/demo-s390x:v0.2
    platform:
      architecture: s390x
      os: linux
  -
    image: s8345002:5000/demo-amd64:v0.2
    platform:
      architecture: amd64
      os: linux

Pushing the manifest list is a simple task then:

root@s8345002:~# manifest pushml demo_v0.2.yml
INFO[0000] Retrieving digests of images...
INFO[0000] Image "s8345002:5000/demo-s390x:v0.2" is digest sha256:9ce1108764ac3573305f0385f48bb13a1dbbab190024b74e537dade5266c9433; size: 1160
INFO[0000] Image "s8345002:5000/demo-amd64:v0.2" is digest sha256:19917b5260edb681a54796ef55ca84de9a25d0847bbebe98fdaaecdd5d01d846; size: 948
Digest: sha256:a8989538239ad5fe35edc7585cdbc3d1ccd17ef073bf5b68b19de6008d2e2548

So now we have assembled s8345002:5000/demo:v0.2 as multi-arch image! We can use it on both s390x and x86 systems in the same way -- here it is on z (the last line is the ID of the started container):

root@s8345006:~# uname -m
s390x
root@s8345006:~# docker run -d s8345002:5000/demo:v0.2
Unable to find image 's8345002:5000/demo:v0.2' locally
v0.2: Pulling from demo
8b882489a3a3: Pull complete
f426950865ca: Pull complete
04b013558d21: Pull complete
30827c9a9c1a: Pull complete
Digest: sha256:a8989538239ad5fe35edc7585cdbc3d1ccd17ef073bf5b68b19de6008d2e2548
Status: Downloaded newer image for s8345002:5000/demo:v0.2
556189a965b6e31b90d867e5f8dcb96ebef4c833e02bb62258b4ee9265e77e5a

on x86 (never mind the z in the name of the host):

root@zhyp223:~# uname -m
x86_64
root@zhyp223:~# docker run -d s8345002:5000/demo:v0.2
Unable to find image 's8345002:5000/demo:v0.2' locally
v0.2: Pulling from demo

e110a4a17941: Pull complete
1d7959f73be3: Pull complete
e402f7bb28d1: Pull complete
Digest: sha256:a8989538239ad5fe35edc7585cdbc3d1ccd17ef073bf5b68b19de6008d2e2548
Status: Downloaded newer image for s8345002:5000/demo:v0.2
8d27377bbd89af5d92448a1986da4f0b690fcb6513b11d3e63b8a291a7249e4f

Different platforms, same commands, same success. q.e.d.! This works for simple "docker run" commands, but also any other tool or orchestration infrastructure that uses docker images: above the docker API level, multi-arch is transparent.

To wrap up, here's how the manifest tool can inspect existing composite images:

root@s8345002:~# manifest inspect s8345002:5000/demo:v0.2
s8345002:5000/demo:v0.2 is a manifest list containing the following 2 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:9ce1108764ac3573305f0385f48bb13a1dbbab190024b74e537dade5266c9433
1  Mfst Length: 1160
1     Platform:
1           -      OS: linux
1           -    Arch: s390x
1           - Variant:
1           - Feature:
1     # Layers: 5
         layer 1: digest = sha256:8b882489a3a36a51c029665ac9cfce0bf9094791321a2d65d78f40709e50bb5c
         layer 2: digest = sha256:f426950865cae44ca253a30e1bd0bcf730b1d300affd793bbcf6aff38ea48cb9
         layer 3: digest = sha256:04b013558d2122c1c979af4934b7d0c6a69fae45dd848fb56a9ee75e3b982f71
         layer 4: digest = sha256:30827c9a9c1aca6526d17f79c4663455e879c87446a5422404487b76702e4bba
         layer 5: digest = sha256:296b0a2fd33c9ec2e1808ad9edfb097d2eb5a94155d6f4503e5fa1fe27bc3e21

2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:19917b5260edb681a54796ef55ca84de9a25d0847bbebe98fdaaecdd5d01d846
2  Mfst Length: 948
2     Platform:
2           -      OS: linux
2           -    Arch: amd64
2           - Variant:
2           - Feature:
2     # Layers: 4
         layer 1: digest = sha256:e110a4a1794126ef308a49f2d65785af2f25538f06700721aad8283b81fdfa58
         layer 2: digest = sha256:1d7959f73be365cad2a5f05cccaf5bdf5169c29cdea4d8b49e3d160f76ac6fab
         layer 3: digest = sha256:e402f7bb28d1d772c50d2abf79d92abbb4f87aef0b3aed39db391c1e1b0dac22
         layer 4: digest = sha256:30a29be64d6006d974ecdef7040346a381a80486e411206faf8aef4e5ef1bce0

Conclusion...

With this technology, you can assemble multi-arch images and store it in a private registry (or Docker Hub). Keeping your images in lockstep for all platforms as you build them and binding them together gives you full flexibility when deploying containers. Re-use is simplified and multiple platforms can be use without pain, always allowing to pick the right platform for the job at the time.