Why We Ship Our OS as an OCI Container

Why We Ship Our OS as an OCI Container

If you’ve ever maintained a fleet of Linux devices in production, you know the dread.

A device in the field starts misbehaving. You SSH in - if you can - and discover it’s running a package version that technically shouldn’t exist. A security patch applied cleanly on 9,998 devices, failed silently on two, and now those two have drifted into a configuration state that no one on your team has ever tested, let alone seen.

This is the default failure mode of every package-based Linux distribution ever shipped. apt, yum, dnf, apk - they’re all imperative systems. You tell them what you want, and they do their best to get the machine to that state from wherever it happens to be right now. The space of “wheres” is effectively infinite.

We decided, very early on, that Admiral would never work this way.

One artefact. Every device. Bit-for-bit identical.

When you deploy an OS release through Admiral, you’re not shipping a list of packages and praying each device converges to the same end state. You’re shipping a single, immutable, signed OCI image that contains the entire root filesystem. Kernel. Userland. Configuration. Your workload. Everything.

Every device pulls the exact same bytes. Every device boots into the exact same state. If the image works on one device, it works on all of them. If it doesn’t work, it doesn’t work uniformly - which, counterintuitively, is an enormous operational win. You can reproduce any production bug on your laptop by running docker pull and docker run.

Drift, as a category of problem, ceases to exist.

Why OCI specifically?

We had three realistic choices for the image format:

1. Roll our own (everyone has tried this; it always ends in tears)
2. Use an existing embedded format like SWUpdate or Mender artefacts
3. Use the OCI image specification

OCI won for a handful of reasons that compound over time:

• Every engineer on Earth already knows how to build one. If you can write a Dockerfile, you can build an Admiral OS image. No new DSL, no new toolchain, no two-week onboarding.
• The ecosystem is staggering. Registries, signing (cosign), SBOM generation (syft), vulnerability scanning (trivy, grype) - all of it works out of the box on an Admiral image because an Admiral image is an OCI artefact.
• Layers are free bandwidth. When you change one file in your rootfs, devices download one layer. A kernel bump doesn’t re-download your application. An application bump doesn’t re-download the kernel. On fleets with 50,000+ devices behind residential ISPs, this matters enormously.
• It’s standards-based. No vendor lock-in. Your images live in your own registry. If you left Admiral tomorrow, you’d still have a working artefact.

“But isn’t a rootfs different from a container?”

This is the question we get most from embedded engineers, and it’s a fair one. A container image is traditionally intended to be run by a container runtime on top of a host OS. We’re using it as the host OS itself.

The trick is that an OCI image is, at the end of the day, just a stack of tarballs with a manifest. There’s nothing in the spec that says those tarballs have to be run inside a namespace. We unpack the layers, flatten them onto an A/B partition, verify the signature, and pivot into them at boot. The kernel, the init system, the entire userland - all of it comes from the image.

What you get is the authoring experience of a Dockerfile with the runtime semantics of a real operating system.

A/B partitions and atomic rollback

Because the image is a single opaque artefact, an update is also a single opaque operation. We write the new image to the inactive partition, verify its signature against your organisation’s root key, flip the boot pointer, and reboot. If the device doesn’t successfully heartbeat back to the control plane within a configurable window, it flips back automatically.

There is no partial update. There is no “apt got halfway through and then the power went out.” The device is either on version N or on version N+1. Nothing else is representable.

What this means for your team

The thing we didn’t fully appreciate until we’d been running this model for a while is the cultural shift it produces inside an engineering team.

When rootfs-es are immutable, your CI pipeline becomes the single source of truth for what runs in production. Nobody SSHes into a device to “just quickly fix something” - they can’t, because any change they made would be wiped on the next update. Every change goes through the image build. Every image build is traceable to a git commit. Every deployment is reviewable in a pull request.

Embedded Linux, for the first time, starts to feel like modern software engineering.

Where we go from here

We’re working on a few things that build directly on this foundation:

• Delta updates at the OCI layer level, for fleets on cellular or satellite links
• Multi-arch manifests so one image tag can resolve to x86_64 or arm64 automatically (already in private beta - more on this soon)
• Provenance attestations baked into every published image, so customers with compliance requirements can prove end-to-end supply chain integrity

If you want to see the model in action, spin up your first five devices for free and push your first image. We think you’ll feel the difference within an afternoon.