chunkah

[!NOTE] This documentation is for the development version of chunkah (quay.io/coreos/chunkah:dev). For the latest stable release, see README.md.

An OCI building tool that takes a flat rootfs and outputs a layered OCI image with content-based layers.

Table of Contents

• Motivation
• Highlights
• Installation
• Verifying image signatures
• Usage
• Splitting an existing image
• Splitting an image at build time
• Advanced Usage
• Understanding components
• Customizing the layers
• Limiting the number of layers
• Output options
• Building from a raw rootfs
• Customizing the OCI image config and annotations
• Pruning and filtering
• Architecture
• Parallelism
• Compatibility with bootable (bootc) images
• Debugging
• Relationship to zstd:chunked
• Origins

Motivation

Traditionally, images built using a Dockerfile result in a multi-layered image which model how the Dockerfile was written. For example, a separate layer is created for each RUN and COPY instructions. This can cause poor layer caching on clients pulling these images. A single package change may invalidate a layer much larger than the package itself, requiring re-pulling.

When splitting an image into content-based layers, it doesn't matter how the final contents of the image were derived. The image is "postprocessed" so that layers are created in a way that tries to maximize layer reuse. Commonly, this means grouping together related packages. This has benefits at various levels: at the network level (common layers do not need to be re-pulled), at the storage level (common layers are stored once), and at the runtime level (e.g. libraries are only mapped once).

chunkah allows you to keep building your image as you currently do, and then perform this content-based layer splitting.

Highlights

• Content agnostic — Compatible with RPM-based images, but not only. Other package ecosystems can be supported, as well as fully custom content.
• Container-native — Best used as a container image, either as part of a multi-staged build, or standalone.
• Zero diff — Apart from modification time, content is never modified.
• Reproducible — Supports SOURCE_DATE_EPOCH for reproducible layers.

It is a non-goal to support initial building of the root filesystem itself. Lots of tools for that exist already. It is also currently a non-goal to preprocess the rootfs to remove common sources of non-reproducibility (such as [add-determinism]). This can be done by the image build process itself.

Installation

chunkah is primarily intended to be used as a container image:

podman run -ti --rm quay.io/coreos/chunkah --help

However, if you're currently building images using a multi-stage build, it may be more convenient to cargo install the binary into your builder image (whether at runtime or build time if you own the builder image). chunkah is also packaged in Fedora, making it easier to do this there.

Verifying image signatures

Container images are signed using [cosign] keyless signing. You can verify the signature of an image using cosign v3+:

cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/coreos/chunkah/' \
  quay.io/coreos/chunkah:latest

Usage

There are two main ways to use chunkah:

• splitting an existing image
• splitting an image at build time

Splitting an existing image

Using Podman/Buildah

When using Podman/Buildah, the most natural way to split an existing image is to use the Containerfile.splitter, passing the target image as the --from:

IMG=quay.io/fedora/fedora-minimal:latest
buildah build --skip-unused-stages=false --from $IMG \
  --build-arg CHUNKAH_CONFIG_STR="$(podman inspect $IMG)" \
  https://github.com/coreos/chunkah/releases/download/v0.6.0/Containerfile.splitter

Additional arguments can be passed to chunkah using the CHUNKAH_ARGS build argument.

[!NOTE] You must add the --skip-unused-stages=false option (see also [this buildah RFE][buildah-rfe]).

For Buildah versions before v1.44, this also requires -v $(pwd):/run/src --security-opt=label=disable. Note that this will leave behind an out/ directory in the current directory.

Another option is using the chunkah image directly and image mounts:

IMG=quay.io/fedora/fedora-minimal:latest
podman pull $IMG # image must be available locally
export CHUNKAH_CONFIG_STR="$(podman inspect $IMG)"
podman run --rm --mount=type=image,src=$IMG,dest=/chunkah \
  -e CHUNKAH_CONFIG_STR quay.io/coreos/chunkah build \
    -t localhost/fedora-minimal-chunked:latest | podman load

The -t/--tag option sets the image name in the OCI archive so that podman load automatically tags the loaded image. Without it, the image is loaded as an unnamed image identified only by its digest.

For large images, this approach is less efficient than using Containerfile.splitter, which avoids the overhead of tarring and untarring the OCI archive (though the same efficiency can be gained by using --output oci:PATH and importing it using skopeo copy.)

Using Docker/Moby

You can use the chunkah image directly using image mounts (requires v28+):

IMG=quay.io/fedora/fedora-minimal:latest
docker pull $IMG # image must be available locally
export CHUNKAH_CONFIG_STR="$(docker inspect $IMG)"
docker run --rm --mount=type=image,src=$IMG,destination=/chunkah \
  -e CHUNKAH_CONFIG_STR quay.io/coreos/chunkah build \
    -t localhost/fedora-minimal-chunked:latest | docker load

Note docker load support for OCI archives requires the [containerd image store] (default on new installations starting from v29+). If using the legacy graph driver, instead of piping directly into docker load as above, you can redirect to a file to save the OCI archive, and then use skopeo to convert to the Docker archive format:

docker run --rm -ti -v $(pwd):/srv:z -w /srv quay.io/skopeo/stable \
  copy oci-archive:out.ociarchive docker-archive:out.dockerarchive:chunked
docker load -i out.dockerarchive

Splitting an image at build time (buildah/podman only)

This uses a method called the "FROM oci: trick", for lack of a better term. It has the massive advantage of being compatible with a regular buildah build flow and also makes it more natural to apply configs to the image, but is specific to the Podman ecosystem. This will not work with Docker.

# Optional; by default base image metadata (like labels) are lost and need to
# either be repeated in the final stage or passed in via a build arg like this
# one. Use with `--build-arg CHUNKAH_CONFIG_STR=$(podman inspect $IMG)`.
ARG CHUNKAH_CONFIG_STR

FROM quay.io/fedora/fedora-minimal:latest AS builder
RUN dnf install -y git-core && dnf clean all

FROM quay.io/coreos/chunkah AS chunkah
ARG CHUNKAH_CONFIG_STR
RUN --mount=type=bind,target=/run/src,rw \
    --mount=from=builder,target=/chunkah,ro \
      chunkah build --output oci:/run/src/out

FROM oci:out
ENTRYPOINT ["git"]

[!NOTE] When building your image, you must also add the --skip-unused-stages=false option (see also [this buildah RFE][buildah-rfe]), and you cannot use the --jobs option.

For Buildah versions before v1.44, this also requires -v $(pwd):/run/src --security-opt=label=disable. Note that this will leave behind an out/ directory in the current directory.

[!NOTE] There is [a known bug][buildah-annotations-bug] in this workflow preventing informational layer annotations added by chunkah from persisting to the final image when additional instructions follow the final FROM. If you're interested in that information, you must either use --config-str instead to pass your config, or run chunkah as a separate step as described in the previous section.

Advanced Usage

Understanding components

A component is a logical grouping of files that belong together. For example, all files from an RPM belong to the same component. Layers are created based on found components.

A component repo is a source of data from which components can be created. For example, the rpmdb is a component repo (one can imagine similar component repos for other distros). There is also an xattr-based component repo (see the section "Customizing the layers" below). Multiple component repos can be active at once.

Customizing the layers

It is possible to create custom components by setting the user.component xattr on files/directories. This can be done using setfattr, e.g.:

RUN setfattr -n user.component -v "custom-apps" /usr/bin/my-app

This is compatible with rpm-ostree's support for the same feature. However, unlike rpm-ostree, this does not guarantee unique layers per xattr component.

In addition, the user.update-interval xattr is also supported. The value is the average number of days between updates, either as an integer or a named label (daily, weekly, biweekly, monthly, quarterly, yearly), e.g.:

RUN setfattr -n user.component -v "custom-apps" /usr/bin/my-app && \
    setfattr -n user.update-interval -v "monthly" /usr/bin/my-app

This only needs to be set on one of the files in the component. If multiple files have conflicting values, chunkah reports an error.

It is strongly recommended to set this xattr. A rough approximation is fine. This helps the packing algorithm make better decisions about which components to group together. When missing, defaults to weekly.

Limiting the number of layers

By default, the maximum number of layers emitted is 64. This can be increased or decreased using the --max-layers option. If the number of components exceeds the maximum, chunkah will pack multiple components together. There is thus a tradeoff in deciding this. Fewer layers means losing the efficiency gains of content-based layers. Too many layers may mean excessive processing and overhead when pushing/pulling the image. Note that containers-storage has a hard limit of 500 layers.

Output options

By default, chunkah writes an OCI archive to stdout. The -o/--output flag controls where and in what format the image is written. It supports an optional transport prefix:

• --output PATH or --output oci-archive:PATH — write an OCI archive to a file instead of stdout.
• --output oci:PATH — write the OCI directory layout directly to disk. This avoids the overhead of tarring and untarring the archive, which is particularly useful in the buildah FROM oci: flow (see Splitting an image at build time).

By default, layers are uncompressed (since in the common case the OCI archive/directory is immediately imported into a container storage backend, which would immediately uncompress it). Use --compressed to enable gzip compression for layers (and the OCI archive itself, if applicable). The compression level can be tuned with --compression-level (0-9, default 6).

Building from a raw rootfs

For completeness, note it's of course also possible to split any arbitrary rootfs, regardless of where it comes from:

podman run --rm -v /path/to/rootfs:/chunkah:z \
  -e CHUNKAH_CONFIG_STR="$(cat config.json)" \
  quay.io/coreos/chunkah build > out.ociarchive

[!NOTE] The :z option will relabel all files for access by the container, which may be expensive for a large rootfs. You can use --security-opt=label=disable to avoid this, but it disables SELinux separation with the chunkah container.

See Output options for controlling the output format and compression.

Customizing the OCI image config and annotations

The OCI image config can be provided via the --config option (as a file) or --config-str/CHUNKAH_CONFIG_STR (inline). The primary format is the [OCI image config] spec as JSON:

{
    "Entrypoint": ["/bin/bash"],
    "Cmd": ["-c", "echo hi"],
    "WorkDir": "/root"
}

The output format of podman inspect and docker inspect are also supported, mostly for convenience when splitting an existing image, though it does also have the advantage of capturing annotations. Otherwise, it's also possible to set annotations directly using --annotation. Labels can also be added via --label.

Pruning and filtering

The --prune option excludes paths from the rootfs. It can be specified multiple times. The trailing slash matters:

• --prune /path excludes the directory and all its descendants entirely.
• --prune /path/ excludes only the contents but keeps the directory itself.

By default, chunkah errors when encountering special file types (sockets, FIFOs, block/char devices). Use --skip-special-files to silently skip them instead.

Architecture

The --arch option overrides the target architecture for the output image. This is useful when splitting an image whose architecture differs from the running host. If not provided, the architecture from the config is used (if available), or the current system architecture otherwise. Common aliases are supported (