Table of Contents

Background

When building and pushing multi-architecture Docker images, you need to be very careful about pushing these images to the registry.

When you push a Docker image to a registry, what happens is that each layer in the image is uploaded as a blob. After every layer is uploaded, the manifest is then updated. The manifest is basically just a JSON file that tells consumer what tags correspond to what blobs.

Pseudocode example:

 1{
 2  "tags": {
 3    "latest": [
 4      {
 5        "platform": "amd64",
 6        "blobs": ["sha256:abcdef"]
 7      }
 8    ],
 9    "2.1.0": [
10      {
11        "platform": "amd64",
12        "blobs": ["sha256:abcdef"]
13      }
14    ]
15  }
16}

So once the layers are finished uploading, the manifest is updated to either add a new tag, or change the existing tag. The thing to be careful of here is that whenever a push is made for an existing tag, the entire contents are overwritten.

Problem

What this means that if you create a multi-architecture build on a single server, everything will be just fine.

 1{
 2    "tags": {
 3        "latest": [
 4            {
 5                "platform": "amd64",
 6                "blobs": [
 7                    "sha256:abcdef"
 8                ]
 9            }
10            {
11                "platform": "arm64/v7",
12                "blobs": [
13                    "sha256:123456"
14                ]
15            }
16        ]
17    }
18}

However, if you split the build across different servers (maybe to take advantage of building an ARM image on an ARM server), whatever build finishes last will be the one overwrites the registry manifest last and will effectively be the only copy available in the registry.

 1{
 2  "tags": {
 3    "latest": [
 4      {
 5        "platform": "arm64/v7",
 6        "blobs": ["sha256:123456"]
 7      }
 8    ]
 9  }
10}

Solution

Building

First, when building the image with docker buildx build, set the following options in the output section:

  • type=image: Export to an image
  • push=true: Still push the image to the registry
  • push-by-digest=true: However, only push the layers, don’t update the manifest
  • name-canonical=true: Add an additional name name@digest. See below.

Additionally, only tag the image with its base name. By this, I mean use --tag docker.io/library/python not --tag docker.io/library/python:3.14.1.

Manifest Update

Now that the image is pushed without the tags, the manifest needs to be updated. Thankfully, this can be done with the docker buildx imagetools create command. The documentation for this command isn’t super obvious, but it takes one or more image name and a list of tags to assign to it.

Be very careful --append flag!!. This will add the selected image(s) to this tag, and not overwrite things. This can easily end up with multiple images for the same platform attached tot he same tag. It’s much safer to add all images at once. For example:

1docker buildx imagetools create -t docker.io/library/python:3.14.1 -t docker.io/library/python:3.14 -t docker.io/library/python:3 docker.io/library/python@sha256:abcdef docker.io/library/python@sha256:ghijkl

By adding the name-canonical=true earlier, you can reference the image you want by the SHA256 digest. Do note that you can only run this command for a single registry. If your image is pushed to more than one registry, then you will need to run this command multiple times.

Conclusion

Hopefully this helps when publishing multi-architecture images with CI/CD. Below is an abbreviated example I’ve made for one of my images using GitHub Actions. I think this is simpler and has less magic than the official Docker example:

  1name: Build and Push Images
  2
  3jobs:
  4  bake:
  5    runs-on: ubuntu-latest
  6
  7    permissions:
  8      contents: read
  9
 10    steps:
 11      - name: Checkout Code
 12        uses: actions/checkout@v6
 13
 14      # Left as an excerise to the reader, but in my case this is a Python
 15      # script that generates a Bake file
 16      # https://docs.docker.com/build/bake/reference/
 17      - name: Create Bake File
 18        run: python dev/baker.py
 19
 20      # Upload the generated Bake file so it can be used across jobs
 21      - name: Upload Bake File
 22        uses: actions/upload-artifact@v6
 23        with:
 24          path: docker-bake.json
 25          name: bake-file
 26
 27      # Splits the Bake file into a build matrix based on the platforms
 28      - name: Generate Build Matrix
 29        id: generate
 30        uses: docker/bake-action/subaction/matrix@v6
 31        with:
 32          # Name of the target in the Bakefile
 33          target: target
 34          fields: platforms
 35
 36      # This is another exercise left to the reader, but this step
 37      # outputs a string with tags that are going to be built prefaced by "-t",
 38      # so it can be piped in to the command later to update the tags in the
 39      # registry after the image is built.
 40      - name: Output Metadata
 41        id: metadata
 42        run: python dev/metadata.py
 43
 44    # Save outputs
 45    outputs:
 46      build-matrix: ${{ steps.generate.outputs.matrix }}
 47      metadata: ${{ steps.metadata.outputs.metadata }}
 48
 49  build:
 50    needs: bake
 51    # Select an ARM runner for ARM platforms
 52    runs-on: ${{ startsWith(matrix.platforms, 'linux/arm') && 'ubuntu-24.04-arm' || 'ubuntu-latest' }}
 53
 54    strategy:
 55      matrix:
 56        # Include the matrix from the previous step
 57        include: ${{ fromJson(needs.bake.outputs.build-matrix) }}
 58
 59    permissions:
 60      contents: read
 61      packages: write
 62
 63    steps:
 64      - name: Checkout Code
 65        uses: actions/checkout@v6
 66
 67      # Download the Bake file so it can be used to build
 68      - name: Download Bake File
 69        uses: actions/download-artifact@v7
 70        with:
 71          name: bake-file
 72
 73      - name: Set up Docker Buildx
 74        uses: docker/setup-buildx-action@v3
 75        with:
 76          version: latest
 77
 78      # Login to the registries we will be pushing to
 79      - name: DockerHub Login
 80        uses: docker/login-action@v3
 81        with:
 82          username: ${{ secrets.DOCKERHUB_USERNAME }}
 83          password: ${{ secrets.DOCKERHUB_PASSWORD }}
 84
 85      - name: Github CR Login
 86        uses: docker/login-action@v3
 87        with:
 88          registry: ghcr.io
 89          username: ${{ github.actor }}
 90          password: ${{ secrets.GITHUB_TOKEN }}
 91
 92      # Actually build the image
 93      - name: Build
 94        uses: docker/bake-action@v6
 95        id: builder
 96        with:
 97          # This ensures we use a local file. Otherwise, it will default to the
 98          # current git repository at the current commit. If your Bake file
 99          # is not generated but static, maybe you want this.
100          source: .
101          targets: ${{ matrix.target }}
102          # Here, we forcefully set the tags to the base image names.
103          # This can also be done in the Bake file generation step.
104          # Additionally, we force the build to a single architecture.
105          set: |
106            *.tags=index.docker.io/username/imagename
107            *.tags=ghcr.io/username/imagename
108            *.platform=${{ matrix.platforms }}
109
110      # By saving the output metadata to a file, it makes it easier for debugging
111      # and referencing in other later steps
112      - name: Save Build Metadata
113        run: echo '${{ steps.builder.outputs.metadata }}' > build-metadata.json
114
115      # Upload artifact for use later
116      - name: Upload Build Metadata
117        uses: actions/upload-artifact@v6
118        with:
119          path: build-metadata.json
120          # A unique name is required as this is a matrix step
121          name: build-metadata-${{ hashFiles('build-metadata.json') }}
122
123  update-digest:
124    needs:
125      - build
126      - bake
127    runs-on: ubuntu-latest
128
129    permissions:
130      contents: read
131      packages: write
132
133    steps:
134      - name: Checkout Code
135        uses: actions/checkout@v6
136
137      - name: Set up Docker Buildx
138        uses: docker/setup-buildx-action@v4
139        with:
140          version: latest
141
142      # Login to the registries we will be pushing to
143      - name: DockerHub Login
144        uses: docker/login-action@v4
145        with:
146          username: ${{ secrets.DOCKERHUB_USERNAME }}
147          password: ${{ secrets.DOCKERHUB_PASSWORD }}
148
149      - name: Github CR Login
150        uses: docker/login-action@v4
151        with:
152          registry: ghcr.io
153          username: ${{ github.actor }}
154          password: ${{ secrets.GITHUB_TOKEN }}
155
156      - name: Download Build Metadata
157        uses: actions/download-artifact@v8
158        with:
159          pattern: build-metadata-*
160
161      # This is another excercise left to the reader. Somehow, this step
162      # needs to known what tags to update in the registries. This can be
163      # potentially be extracted from the Bake file or by some other means.
164      # However, it cannot come from the `builder` step, because that output will
165      # not have the tags since it was intentionally stripped to ensure
166      # `push-by-digest` works.
167      # Additionally, it needs to know the image digests, though it could extract
168      # those from the downloaded JSON files from the previous step.
169      - name: Upload Digest
170        run: python dev/imagetools.py ${{ fromJSON(needs.bake.outputs.metadata).tags }}

Obviously, there are other ways you can approach this depending on how you tag images, whether or not you commit a Bake file, etc., but I’m found this approach to work well for me for what I’m doing. It also manages to skip the final “merge” step all of the official examples show which adds complexity.

References