Some time ago, our team decided to deploy the application which we are developing for our customer as a docker container. As docker is a promising but still very young technology, this decision naturally put us on a quest for finding a reliable, secure and maintainable setup — many things are still in flux in the community, and the resulting lack of proven best practices leaves a lot of room for experiments (sometimes frustratingly so).

In this blogpost, I want to share one result of our experiences: how to set up and maintain a secure private docker registry.

Why should I set up a private registry?

Storing images. Lots of them.

If you want to run a reasonably complex web application on docker, you will have to build a lot of docker images. In our case, we have at least the application itself, a CLI image (providing a maintenance shell), a database image (that replaces the DB in our development environment), a reverse proxy for development, another reverse proxy (switching between containerized application versions for zero-downtime deployment) and build container that hosts our build process. On top, we also have a bunch of common ancestor containers. In total, we have more than ten images that we maintain and build.

All these images must be stored somewhere. Rebuilding them from scratch on each machine that uses them is not feasible, and as this is a closed source project, putting them on the public docker hub is not an option either. A private registry gives us a nice, secure place to store our images that seamlessly integrates with docker.

Reproducible base images

All our containers eventually descend from the official docker hub Alpine and Debian Linux containers. In order to maintain a degree of reproducibility in the build process, we’d like to be able to target a fixed version of the base image. However, the official image tags on the docker hub are moving targets, and the underlying versions change as distribution updates are published. In order to maintain reproducible base images, we have our own base images that inherit from the official docker hub images and which we store in our own registry. In addition, building our images once and storing them in our registry ensures that we can always access the exact images that we deployed to production.

Deployment

Running our own docker registry makes deployment to production as simple as „docker pull“ and then starting the container (we use crane for container management).

Running the registry

While the full docker hub is closed source, the registry („docker distribution“) behind it is open source, written in Go and hosted on github. Unsurprisingly, the official way to run it is a docker image. Nothing special here, and official documentation for running the image is pretty exhaustive. Some remarks:

Configuration

The registry offers a plethora of configuration options that are supplied as a YAML config file. If you want to change the default configuration (you better should), you can either overwrite individual settings using environment variables or package your own config into a container that inherits from the docker hub image. Our corresponding Dockerfile looks like

FROM library/registry:2.3.0

COPY config /config
RUN mkdir /data
 
VOLUME /data
CMD ["/config/config.yml"]

A simplified version of our configuration is

version: 0.1
loglevel: debug
storage:
  filesystem:
    rootdirectory: /data
  delete:
    enabled: true
 
http:
  addr: :5000

Storage

There is a number of storage options to choose from, including cloud-based options. However, the default file storage is sufficient for our use case. Just make sure that you have lots of storage and mount it into the container (this ensures that your storage can be easily backed up and will survives updates).

SSL

Securing the registry via SSL is a good idea. While the registry supports SSL (provided you supply a certificate), you can also run the registry over HTTP and use a reverse proxy to offload SSL decryption instead. This is particularly useful if the registry is not the only web service running on the host.

Security

As the registry is the hinge pin of our build and deployment process, securing it is vital. By default, everybody who can access the registry server has full permissions for reading and writing images. The registry offers two options for securing its content: HTTP basic auth and a custom token-based authentication protocol. Basic auth is simple to set up and use, but does not allow for any kind of permission management: all authorized users have full access to the registry. The second option is more complicated, but offers more way more flexibility.

Token-based authentication protocol

The token-based authentication protocol is described in detail here and involves both the registry and a authentication server. What happens is basically this: on accessing the registry, an unauthorized client is presented with a challenge that includes the requested resource and action together with the URL of an authentication server. The client then contacts the authentication server and authenticates (via basic auth). If access is granted for the requested resource and action, the auth server responds with a cryptographically signed access token. Using this token, the client contacts the docker registry again. The registry validates the token and, if the signature is valid, proceeds with the requested action.

This protocol allows to restrict user permissions an ACL. However, while the protocol is documented, there is no open source reference implementation of the actual auth server by Docker Inc.

Auth server implementations

While the official auth server is not public, there are at least two projects implementing this gap in the spec. Among these, we settled on docker_auth: it is simple to set up and deploy (being written in Go) and offers the option to configure a simple list of users and ACL rules in a static configuration file (more complex configuration schemes are supported as well).

Another option is portus. While portus is much more ambitious and also offers a GUI for browsing and user managerment, we did not get it to work reliably. However, the project is promising and is absolutely worth a try — who knows, it might work better for you than it did for us.

Setting up docker_auth

The natural way to run docker_auth in this setup is a docker container. A simplified version of our Dockerfile looks like this:

FROM alpine:3.3
 
RUN apk add --update coreutils curl
 
RUN adduser -D dockerauth \
  && apk add --update -t build-group git go wget \
  && GOPATH=/tmp go get github.com/cesanta/docker_auth/... \
  && mv /tmp/bin/auth_server /auth_server \
  && rm -rf /tmp/* \
  && apk del build-group \
  && rm -rf /var/cache/apk/*
 
COPY config /config
 
RUN chown -R dockerauth: /config
USER dockerauth
EXPOSE 5001
 
CMD ["dockerauth", "/auth_server", "-logtostderr", "/config/config.yml"]

Similar to the registry, docker_auth reads its configuration from a YAML file. The most important aspects of the configuration are:

• The certificate used for signing the issued access token. The same certificate must be configured with the registry for verifying the signature. The certificate used for this purpose may be self-signed.
• A list of users with their corresponding bcrypt hashed passwords
• A set of ACL rules

As a reference, our corresponding config roughly looks like this

server:
  addr: ":5001"
 
token:
  issuer: "ACME auth server - aa8AhshuoCh5eade"
  expiration: 900
  certificate: "/config/cert.pem"
  key: "/config/private.pem"
 
users:
  admin: bcrypt_hashed_admin_password
  readonly: bcrypt_hashed_read_only_password
  someuser: bcrypt_hashed_stuff_password
 
acl:
  - match:
      account: "admin"
    actions: ["*"]
    comment: "Admin has full access to everything."
  - match:
      account: "readonly"
    actions: ["pull"]
    comment: "Read only access."
  - match:
      account: "someuser"
      name: "someuser/*"
    actions: ["*"]
    comment: "User can access his own namespace"

This example configures three users: admin has full registry access, readonly has full readonly access to the registry, and someuser can only access their own repositories (prefixed with someuser/).

Configuring the docker registry

Once docker_auth is up and running, we must tell the registry how to use it. The corresponding part of the config looks like this:

auth:
  token:
    realm: https://foo.bar.com/auth
    service: docker.foo.bar.com:443
    issuer: "ACME auth server - aa8AhshuoCh5eade"
    rootcertbundle: /config/cert.pem

Important aspects of the configuration:

• Realm is the URL for contacting the auth server
• Issuer must match the corresponding setting configured for docker_auth
• The certificate is the same cert configured with docker_auth for signing the tokens

Using the registry with docker

Once we have both the registry and docker_auth up and running, it is time to try and access it with docker. Before we can use our registry, we have to tell docker about it (assuming the registry is hosted on docker.foo.bar.com):

docker login docker.foo.bar.com

Docker will now ask for username and password. Be careful: docker will store the credentials unencrypted in a JSON file in your home directory.

After „logging in“, the registry can be used by prefixing a repository with the registry’s fully qualified domain name:

docker tag someimage docker.foo.bar.com/some/repository:1.0
docker push docker.foo.bar.com/some/repository:1.0

Browsing the registry with docker-ls

While the official docker hub can be easily browsed from the web, the open source docker registry lacks this functionality. As there is currently no tool replacing this functionality, we created our own: docker-ls. This tool allows you to browse the contents of a docker registry on the CLI and supports authentication both via basic and via token-based auth. Some short examples:

List all repositories:

docker-ls repositories --registry https://docker.foo.bar.com \
   --user <username> --password <password>

List all repositories, including tags:

docker-ls repositories --registry https://docker.foo.bar.com \
   --user <username> --password <password> --level 1

List all tags in a repository, including content digests:

docker-ls tags --registry https://docker.foo.bar.com \
   --user <username> --password <password> --level 1 some/repository

Show manifest for a single tag

docker-ls tag --registry https://docker.foo.bar.com \
   --user <username> --password <password> --level 1 --raw-manifest some/repository

The project also includes docker-rm, a tool for deleting individual tags from the registry For more documentation, head over to the github page.