How hash IDs are managed for each RUN statements in Dockfile?

Question

For example, I may have the following Dockfile. When I run docker build, for each RUN, there is a spearate hash (e.g., 1d9c17228a9e), and it runs very fast it had run already. I guess each hash is associated an actual file at the backend. Is it so?

If there are separate files, how they can be loaded in a single virtual machine quickly? Is there any kind of assemble upon starting a new virtual machine (docker container)? Thanks.

$ docker build -t ubtsrv .
Sending build context to Docker daemon  12.29kB
Step 1/22 : FROM ubuntu
 ---> 1d9c17228a9e
Step 2/22 : RUN rm -rf /etc/dpkg/dpkg.cfg.d/excludes
 ---> Using cache
 ---> eb02f606ba08
Step 3/22 : RUN apt-get -y update &&     dpkg -l | grep ^ii | cut -d' ' -f3 | xargs apt-get install -y --reinstall
 ---> Using cache
 ---> 7062816b0023
Step 4/22 : RUN apt-get -y install apt-utils
 ---> Using cache
 ---> b89d4cdb791c
Step 5/22 : RUN apt -y update && apt -y upgrade
 ---> Using cache
 ---> 8100af2b7f2e
Step 6/22 : RUN apt-get -y install vim
 ---> Using cache
 ---> 57c142f99935
Step 7/22 : RUN apt-get -y install man
 ---> Using cache
 ---> ddb73e4bbddc
Step 8/22 : RUN apt-get -y install gawk
 ---> Using cache
 ---> 7422b4371c16
Step 9/22 : RUN apt-get -y install mawk
 ---> Using cache
 ---> 53a01709a342
Step 10/22 : RUN apt-get -y install build-essential
 ---> Using cache
 ---> af94947e6922
Step 11/22 : RUN apt-get -y install command-not-found
 ---> Using cache
 ---> 20094698a583
Step 12/22 : RUN apt-get -y install clang
 ---> Using cache
 ---> e63570058a57
Step 13/22 : RUN apt-get -y install htop
 ---> Using cache
 ---> b09fec30dc23
Step 14/22 : RUN apt-get -y install wget
 ---> Using cache
 ---> d2794d29f9ee
Step 15/22 : RUN apt-get -y install curl
 ---> Using cache
 ---> 2b122c49f3ca
Step 16/22 : RUN wget -q ftp://ftp.gnu.org/gnu/bash/bash-4.4.18.tar.gz &&   tar xzvf bash-4.4.18.tar.gz &&   cd bash-4.4.18 &&   ./configure &&   make -j &&   make install &&   cd .. &&   rm -rf bash-4.4.18.tar.gz bash-4.4.18
 ---> Using cache
 ---> c4bf046aff2a
Step 17/22 : RUN apt-get install -y git
 ---> Using cache
 ---> 40ebefa7acda
Step 18/22 : RUN apt-get install -y ack
 ---> Using cache
 ---> 05cefb3f0496
Step 19/22 : RUN apt-get install -y info
 ---> Using cache
 ---> 3361e4e4e06f
Step 20/22 : RUN apt-get install -y llvm
 ---> Using cache
 ---> 50b7c75fc2f5
Step 21/22 : RUN apt-get install -y graphviz
 ---> Using cache
 ---> 80f89477930c
Step 22/22 : RUN apt-get install -y cmake
 ---> Using cache
 ---> c8320b1b2523
Successfully built c8320b1b2523
Successfully tagged ubtsrv:latest
$ cat Dockerfile 
FROM ubuntu
RUN rm -rf /etc/dpkg/dpkg.cfg.d/excludes

RUN apt-get -y update && \
    dpkg -l | grep ^ii | cut -d' ' -f3 | xargs apt-get install -y --reinstall
RUN apt-get -y install apt-utils
RUN apt -y update && apt -y upgrade
RUN apt-get -y install vim
RUN apt-get -y install man
RUN apt-get -y install gawk
RUN apt-get -y install mawk
RUN apt-get -y install build-essential
RUN apt-get -y install command-not-found
RUN apt-get -y install clang
RUN apt-get -y install htop
RUN apt-get -y install wget
RUN apt-get -y install curl
RUN wget -q ftp://ftp.gnu.org/gnu/bash/bash-4.4.18.tar.gz && \
  tar xzvf bash-4.4.18.tar.gz && \
  cd bash-4.4.18 && \
  ./configure && \
  make -j && \
  make install && \
  cd .. && \
  rm -rf bash-4.4.18.tar.gz bash-4.4.18
RUN apt-get install -y git
RUN apt-get install -y ack
RUN apt-get install -y info
RUN apt-get install -y llvm
RUN apt-get install -y graphviz
RUN apt-get install -y cmake

Answer 1

Each hash is a docker layer. It's just a filesystem layer containing the different files added in that step. If you dip into docker internals you can actually take a look at the specific files that were added.

This section on docker caching describes how docker decides what is cached and what is not: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#leverage-build-cache

This tool is a lot of fun: https://github.com/wagoodman/dive an easy way to explore your docker images and check out the contents of each layer.

Let's talk through an example dockerfile:

FROM alpine

WORKDIR /opt/
RUN touch foo && mkdir bar && touch bar/foo

RUN rm foo && touch file.txt

RUN rm -rf bar

Here's the build output:

Building app
Step 1/5 : FROM alpine
 ---> 196d12cf6ab1
Step 2/5 : WORKDIR /opt/
 ---> Running in 2098e27c28b9
Removing intermediate container 2098e27c28b9
 ---> 74634b6a7dcd
Step 3/5 : RUN touch foo && mkdir bar && touch bar/foo
 ---> Running in f109a620ebfd
Removing intermediate container f109a620ebfd
 ---> dea70d465cc1
Step 4/5 : RUN rm foo && touch file.txt
 ---> Running in 367e61e301ba
Removing intermediate container 367e61e301ba
 ---> 9dcca4810268
Step 5/5 : RUN rm -rf bar
 ---> Running in d176de336110
Removing intermediate container d176de336110
 ---> 2e2eee6b9bf8

Successfully built 2e2eee6b9bf8
Successfully tagged docker-fsl_app:latest

If I run docker inspect 2e2eee6b9bf8 (the outputed hash above) docker returns a bunch of data. Included in that are two sections:

"GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/de87e6b38f95b44137409b5a61b498781473bc05cfd74a01dd641245219c2a1f/diff:/var/lib/docker/overlay2/02d58096fd47908c82edbc34dd0205541e525afe804e88f517ff47ccf3beeee0/diff:/var/lib/docker/overlay2/91fb3592a0da4847071a51e7dda4f48b810a5d1ff0b22e34bb38a0ee52d13d09/diff:/var/lib/docker/overlay2/2e966b19c5984548a6adb172d092dd21b2bb73f6be839baa680dc524d5221063/diff",
                "MergedDir": "/var/lib/docker/overlay2/3216972ae99360398a74720226b26b61f0c04142ad6aaa519c1a9dd36f7fb945/merged",
                "UpperDir": "/var/lib/docker/overlay2/3216972ae99360398a74720226b26b61f0c04142ad6aaa519c1a9dd36f7fb945/diff",
                "WorkDir": "/var/lib/docker/overlay2/3216972ae99360398a74720226b26b61f0c04142ad6aaa519c1a9dd36f7fb945/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
                "sha256:b9f91d14f5d797f43eeb5b56264cc697641d50dd5e9d17bf89f33cf0694f6559",
                "sha256:97195b4b7c22c7eb8720edeb93feeb6901a34018ce1f3c90dc17f861438abf21",
                "sha256:0f3d56ac5865b537686b1e324dfbf54edde5afd06e644903ad6b9af42eab01df",
                "sha256:5ff5ef92db130446e0af4836ffba8fbf29d06643aa05a104cb4c7a4c9e462fc7"
            ]
        },

I'm on osx. On osx I can run screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty to access the docker vm. Within the vm, I can actually look at the filesystem layers.

These are the layers in reverse order: /var/lib/docker/overlay2/de87e6b38f95b44137409b5a61b498781473bc05cfd74a01dd641245219c2a1f/diff:/var/lib/docker/overlay2/02d58096fd47908c82edbc34dd0205541e525afe804e88f517ff47ccf3beeee0/diff:/var/lib/docker/overlay2/91fb3592a0da4847071a51e7dda4f48b810a5d1ff0b22e34bb38a0ee52d13d09/diff:/var/lib/docker/overlay2/2e966b19c5984548a6adb172d092dd21b2bb73f6be839baa680dc524d5221063/diff

If you go to those locations you'll see just the files added or removed in that layer. So if I go to /var/lib/docker/overlay2/de87e6b38f95b44137409b5a61b498781473bc05cfd74a01dd641245219c2a1f/diff/opt within the vm and run ls -lah. This is the output:

drwxr-xr-x    2 root     root        4.0K Jan 14 16:15 .
drwxr-xr-x    3 root     root        4.0K Jan 14 16:15 ..
-rw-r--r--    1 root     root           0 Jan 14 16:15 file.txt
c---------    1 root     root        0,   0 Jan 14 16:15 foo

file.txt has been added and foo has been deleted (I think that's why foo doesn't have permissions, the specific details of what a "deleted" file is is unclear to me).

So for every build layer the diff of files added or deleted is added as a lyaer.