Cache-friendly docker images

This is follow up post after my latest blogpost about bringing separate docker layers for application dependencies, configuration and actual code. Despite the previous post was about building docker images for Scala application this one should be more general. I invite to read everyone taking care about size of their docker images in the CI environment.

Reality

Did grouping artefacts to layers worked? In local environment it did, but not in the real world (at least not without additional configuration).
I was presented example showing docker cache is not reused. After studying docker for more time it appeared that building docker images is not deterministic and depends on machine time.

Digest differs for same images if cache cannot be used

One cannot simply build same docker image twice

Reproducible builds

Reproducible builds happen when you’re able to build exactly same artifact on some other machine. Definitely docker image builds are NOT reproducible. The docker build timestamp is preserved directly into docker layer config JSON files under created property. The digest of image layer is based on checksum of files including their content, creation&modification timestamps and permissions. Some of the files JSON files represents docker layer metadata and count to digest calculation. Very good explanation could be found in this blogpost.

Reproducible builds deserved even own manifest reproducible-builds.org

JIB

The Google JIB plugin that was motivation behind adding docker layer grouping to SBT native plugin is able to create reproducible builds. You can stop reading and just use SBT JIB plugin. How do they do it?

• They set file deterministic file attributes (timestamps=EPOCH+1 second, permissions, owner) before adding it to the image.
• They set docker layer creation timestamp to epoch (1970–01–01T00:00:00Z).

Great, can we do it in SBT? Unfortunately not, docker build api does not have such parameter. You may ask how they did it? They’ve just build docker compiler which is packaging *.jars to *.tar layer files together with java generated JSON configuration&manifest to match docker image standard. Docker image is in the end just collection of tar archives.

I’m not against such approach as it’s battle tested by huge company. However I‘d prefer to stick to mainstream tool to build images. I’ve learned there exists BuildKit — the next generation backend for creating container images that is separated from Dockerfile standard. The issue to add more control over layer timestamps already exists.

BuildKit

Ok, it’s all very interesting, but you’ve all came here for optimising storage of your docker repositories. Let me present you two workarounds. First one is based on alternative, IMHO soon to be mainstream docker image compiler Buildkit. You should use it if your CI server runs at least docker 19.03.x. Then try if it works, as some of cloud CI providers and docker registries might not yet be compliant (like mine). Use second approach only if BuildKit doesn’t work for you.

Concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit.
BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner.

I used to avoid publishing docker images with latest tag. But good exception is to publish your docker cache. The image becomes dangling if you tag other one with latest tag and hosted images repository should have easy task to clean it up. You may consider tagging you cache images with your git branches.

docker 19.03.x

Unfortunatelly the ecosystem has not adapded BuildKit yet. Despite it works for me locally, i’ve already encountered problems in my production evironments. The JFrog image upload script has some failures. Also my CI server cannot read cache failing with error on cache query: invalid build cache from.

No BuildKit

If for some reason like me you cannot adopt Buildkit, here is the workaround. Docker build has various issues with using cache together with multistage images. The workaround is to build base stage and final image in two separate commands.

CI

Future

The main idea is to bring real reproducible builds to Scala and SBT. Unlike in JIB the plan is to still build images using regular docker build . tool.
The steps are already defined.

• Deterministic file creation&modification timestamps sbt-native-packager#1321
• Better cache reuse from multistage build sbt-native-packager#1325
• Deterministic docker layer creation timestamp when building with BuildKit buildkit#1058 (please vote for it)

Having that --cache-from will not be necessary if you optimize only for docker repository space. However the cache still can bring you benefit in speed of building the image.

Hope that reading this post helped you to discover that your Docker CI uses less caching than you expected. The simple configuration from snippets