The hidden design compromises of Docker layers

The other week I was looking into some Docker shenanigans, specifically SOCI (Seekable OCI). In a nutshell, SOCI builds an index of the contents of your container image layers, so that a container can start before the whole image has been downloaded, and the files it needs are lazily fetched as they are accessed. (Spoiler: there will probably be an AWS Bites episode about SOCI soon, so stay tuned there if you are curious.) While reading about compressed layers, indexes and lazy loading, I started poking at my own understanding of Docker images. And I didn’t love what I found. I have been using Docker for years. I could happily tell you that “an image is made of a stack of immutable layers” and I would probably even draw you a nice diagram with some boxes stacked on top of each other. But if you asked me what those layers actually contain, my answer would get hand-wavy pretty quickly. One question in particular got stuck in my head: If Docker layers are basically tar archives applied…