This episode will consist of a quick aside to build a Docker Registry Mirror.
Previous: Part 1
Why Do We Need a Mirror??
The Docker Registry is, to my mind, one of the greatest contributors to the rapid growth of Docker. Having a central location for images encourages sharing and re-use. Obviously some images are more useful than others, but it is a wonderful resource for learning and leveraging the work of others. I like to look at Dockerfiles written by others; I learn a lot that way. It also helps me to determine which image to use where there are more than one providing a service. (Hint: I prefer to avoid those who do not share their Dockerfile).
In the Raspberry Pi Swarm, there are containers which will be pulled by all of the hosts. It takes a while to pull down images, even in parallel (at which point network congestion can cause problems). If an image is 300MB, unless we have a local mirror the entirety of the image is going to be pulled down across the internet by each host. It’s a lot quicker to cache the image locally and pull it once. Also, without a local mirror, there is a dependency on internet connectivity — something I am attempting to avoid with my Pi Swarm.
Configuring the Mirror Host
Requires:
1. Raspberry Pi
2. Hypriot Image (or other image supporting recent Docker)
3. External Disk (see below)
As before, I am starting from a Hypriot install. I have attached an external disk (in my case I’m using a SSD with LVM) to /opt/docker-cache to serve as my mirror. I’m not sure that using a micro-sd for the cache is the best way to go — it’s slow, for one thing. I’m not certain that the benefit of a less internet traffic and latency is overcome by the slow disk.
I’ve uploaded a Docker Registry image for the Raspberry Pi to the Docker Hub. You can pull it via:
|
1 2 |
docker pull nimblestratus/rpi-docker-registry |
The Dockerfile follows — there are a few differences from the stock Dockerfile:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
# VERSION 0.1 # DOCKER-VERSION 0.7.3 # ORIGINAL AUTHOR:Sam Alba <sam@docker.com> # RASPBERRY PI: Matt Williams <matt@matthewkwilliams.com> # DESCRIPTION: Image with docker-registry project and dependecies # TO_BUILD: docker build -rm -t registry . # TO_RUN: docker run -p 5000:5000 registry FROM hypriot/rpi-python # Update RUN apt-get update # Install pip && apt-get install -y swig build-essential python-pip # Install deps for backports.lzma (python2 requires it) python-dev python-mysqldb libssl-dev liblzma-dev libevent-dev && rm -rf /var/lib/apt/lists/* COPY . /docker-registry COPY ./config/boto.cfg /etc/boto.cfg # python-rsa run pip install rsa # boto run pip install boto # Install core RUN pip install /docker-registry/depends/docker-registry-core # Install registry RUN pip install file:///docker-registry#egg=docker-registry[bugsnag,newrelic,cors] RUN patch $(python -c 'import boto; import os; print os.path.dirname(boto.__file__)')/connection.py < /docker-registry/contrib/boto_header_patch.diff ENV DOCKER_REGISTRY_CONFIG /docker-registry/config/config_sample.yml ENV SETTINGS_FLAVOR dev EXPOSE 5000 CMD ["docker-registry"] |
If you want to build it yourself, you can do the following (on a pi with Docker):
|
1 2 3 4 5 |
git clone git://github.com/nimblestratus/docker-registry cd docker-registry docker build -t rpi-docker-registry . docker tag rpi-docker-registry nimblestratus/rpi-docker-registry |
Running the Mirror
Once you’ve built (or pulled) the registry image, it can be started on the registry host. In order for the mirroring to work properly, modifications to the other Docker hosts will need to be made. That follows below.
I am assuming that the external disk is mounted at: /opt/docker-cache.
|
1 2 3 4 5 6 7 8 |
docker run -p 80:5000 -v /opt/docker-cache:/tmp/registry -d -e STANDALONE=false -e MIRROR_SOURCE=https://registry-1.docker.io -e MIRROR_SOURCE_INDEX=https://index.docker.io -e GUNICORN_OPTS=["--preload"] nimblestratus/rpi-docker-registry |
There are a couple arguments of interest:
| Argument | Notes |
|---|---|
-p 80:5000 |
The registry is exposed on port 80. This serves two purposes. First, if you wish to have a local registry of private images running on host registry, you can specify registry/project/image instead of registry:5000/project/image when pushing or pulling an image. Additionally, should you need to delete an image, docker is (at the moment) unable to delete images of the form HOST:PORT/PROJECT/IMAGE. |
-v /opt/docker-cache:/tmp/registry |
This points to the location outside of the container which is used for the mirror’s cache. If you don’t specify a location, it’s kept within the container and once the container ends you’ve lost the benefits of a mirror. |
-e GUNICORN_OPTS=["--preload"] |
This is to workaround a bug. The registry will not start up correctly without it. See Issue #892 · docker/docker-registry for a discussion of it. |
Using the Mirror
Docker hosts which you wish to use the mirror need extra arguments passed into the daemon upon start. Assuming you are running a debian based version, the default configuration file is located at /etc/default/docker.
The DOCKER_OPTS variable needs to be edited to add --registry-mirror=http://REGISTRY_MIRROR. Replace REGISTRY_MIRROR with either the name or IP address of the host on which the mirror is running.
Once the file is edited, restart docker:
sudo service docker restart
To test, pull in a docker image you’ve not pulled before, remove it, then pull in again: (you may briefly see a message saying it’s using the mirror)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
matt@argentum:~$ time docker pull hypriot/rpi-java Pulling repository hypriot/rpi-java 2fdcc54418d3: Download complete edb6a2ed6796: Download complete 190765ac354b: Download complete 6efda10ba849: Download complete 7dae865b8532: Download complete 4128d6964db0: Download complete 7203404fb0de: Download complete Status: Downloaded newer image for hypriot/rpi-java:latest real 0m43.763s user 0m0.032s sys 0m0.042s matt@argentum:~$ docker rmi hypriot/rpi-java Untagged: hypriot/rpi-java:latest matt@argentum:~$ docker images |grep rpi-java hypriot/rpi-java jre-1.7.0 2fdcc54418d3 5 weeks ago 210.2 MB matt@argentum:~$ docker rmi hypriot/rpi-java:jre-1.7.0 Untagged: hypriot/rpi-java:jre-1.7.0 Deleted: 2fdcc54418d39b8fd1029420e01276405c3d7d0eb2c2d19113278092a182975b Deleted: 7203404fb0deb3551db2a322e7ff51ce807d5baa9feb1e70626a5b852359f5b6 Deleted: 4128d6964db0ec957a2fc147605b49ff5911bc3248d31424ea815fcb8bcee97f Deleted: 7dae865b8532160e8ea03ae2590a2531e6091d936e15cbe0a9fdd30a69551873 matt@argentum:~$ time docker pull hypriot/rpi-java Pulling repository hypriot/rpi-java 2fdcc54418d3: Download complete edb6a2ed6796: Download complete 190765ac354b: Download complete 6efda10ba849: Download complete 7dae865b8532: Download complete 4128d6964db0: Download complete 7203404fb0de: Download complete Status: Downloaded newer image for hypriot/rpi-java:latest real 0m15.997s user 0m0.057s sys 0m0.019s matt@argentum:~$ |
We went from ~44 to ~16 seconds total, almost 1/3 the time.
Limitations
Ultimately the limiting factor for improvement is I/O, namely the speed of the USB Bus and the bandwidth of the local network. In this case, the mirror is plugged into a 10/100Mb switch. The host performing the pull is connected over wireless. Kidzilla is watching Netflix.
As a quick test of network speed:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
$ time dd if=/dev/zero bs=64K count=1000 | ssh matt@argentum "cat >/tmp/fud" 1000+0 records in 1000+0 records out 65536000 bytes (66 MB) copied, 16.4853 s, 4.0 MB/s real 0m17.728s user 0m7.380s sys 0m3.600s $ scp -r 9ef2196a52918bff52d95ffd56c007fb052e8e8506a33734386df099b399df58 matt@argentum:/tmp layer 100% 76MB 4.8MB/s 00:16 json 100% 1587 1.6KB/s 00:00 |
Ok, so in this totally unscientific test, I can fill 48Mb/s. Let’s try the same, but wired:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
$ scp -r 9ef2196a52918bff52d95ffd56c007fb052e8e8506a33734386df099b399df58 matt@nimbus:/tmp layer 100% 76MB 5.9MB/s 00:13 json 100% 1587 1.6KB/s 00:00 $ time dd if=/dev/zero bs=64K count=1000 | ssh matt@nimbus "cat >/tmp/fud" 1000+0 records in 1000+0 records out 65536000 bytes (66 MB) copied, 12.7135 s, 5.2 MB/s real 0m12.762s user 0m7.280s sys 0m4.980s |
A little bit better; so at this point either the 10/100Mbs switch is being flooded or we’ve hit the limit of the USB bus. There are other processes running on the Pi, too, some of which are chatty.
Next Time
Next time is using consul and registrator along with swarm. If you’d like to play in the mean time, the images are out on docker hub:
| Service | Image | Notes |
|---|---|---|
| consul | nimblestratus/rpi-consul | Port of progrium/consul |
| registrator | nimblestratus/rpi-registrator | Port of gliderlabs/registrator |
| swarm | nimblestratus/rpi-swarm | Port of docker/swarm. Details on how to use it at: Swarming Raspberry Pi Part 1 |
4 comments
2 pings
Skip to comment form ↓
Dieter Reuter (@Quintus23M)
April 4, 2015 at 10:08 am (UTC -5) Link to this comment
Matt, you’re using a SSD connected to the RPi. Could you just give some details on how to attach the SSD and maybe some performance comparison between SD cards and SSD in your setup? I’d like to get an idea how big the improvement is, one could gain when using a SSD attached via USB2.0.
Matt Williams
April 4, 2015 at 4:52 pm (UTC -5) Link to this comment
Definitely.
I’ll work on that tonight.
I’m just about finished with writeup on bonding NICs. I didn’t see much difference (in the artificial test I did) with the B+, but defintitely difference on the Pi 2B.
And discovered something potentially ugly about cron on the B+ with the Hypriot version from the beginning of March… The short form is that cron is eating almost 1/2 the CPU…. even with no crontabs.
Steve Woolley
April 15, 2015 at 12:50 pm (UTC -5) Link to this comment
Hi Matt,
Spelling on the rip-consul image table wrong on blog post. Should be “nimblestratus”. Only way I noticed was tried to cut and past this into docker hub search bar.
Thanks for the excellent posts.
Matt Williams
April 15, 2015 at 1:00 pm (UTC -5) Link to this comment
Thanks for the catch!
I’ll fix it right away.
I’ve been having lots of fun with them, truth be told!
Swarming Raspberry Pi: Private Registry for Swarm Images » Ramblings
April 15, 2015 at 12:05 am (UTC -5) Link to this comment
[…] need to have a registry running. Swarming Raspberry Pi Part 2: Registry and Mirror has […]
350+ ???????? ????????, ???? ? ???????????? ??? ?????? ? Docker — ? ???? ??
January 14, 2016 at 4:14 am (UTC -5) Link to this comment
[…] Matthewkwilliams: ????????????? Swarm ??? Raspberry Pi – ????????? ? ???????; […]