Swarming Raspberry Pi: Private Registry for Swarm Images

Some more backstory on the Pi Swarm

I was really excited when Amazon announced their Lambda offering. I thought that it was an awesome idea, but for the lack of an open solution and that it locked you into javascript.

I believe that using Docker, we can have a relatively simple Amazon Lambda work-alike which allows code from arbitrary languages to be run.

Along the way, I’ve investigated using Kubernetes, but it didn’t support ephemeral containers. It kept trying to resurrect the dead container. Hilarity ensued after a fashion.

Enter Swarm….

Swarm

I’d seen mentioned that Swarm wasn’t working with servers which require authentication. Issue #374, in its history, indicates that there is no (as of yet) support for registry requiring authentication.

It occurred to me that it might be possible to have swarm working with a local private registry via an insecure registry. A few tests later, and it’s alive!!!

Configuration

Registry

You’ll need to have a registry running. Swarming Raspberry Pi Part 2: Registry and Mirror has details.

Private registries have the standalone=true flag set. According to the documentation:

On first reading, it looks as though a registry cannot work as both a mirror and a private registry. In tests, however, I was able to use a private registry as both a mirror and private registry. I believe but have not verified that the registry is passing index queries to the docker hub. I have verified that images cached on the local private registry are served locally. So… it might be thought of as a private registry which just so happens to act as a image cache for images from the canonical registry. However, it is not indexing the images. If you care to experiment, you can start a registry as follows:

On a Raspberry Pi, substitute nimblestratus/rpi-docker-registry for the image. One thing which I didn’t get working (although didn’t test extensively — the TCP/IP stack on my laptop gets fussy after repeatedly connecting to different networks and connecting/disconnecting from multiple VPNs) is to run a mirror registry as a docker container and pointing the docker daemon to the registry container. Part of me thinks it might work, but I can also see where it wouldn’t — Docker might attempt to talk to the registry on startup, realize it isn’t up, then give up. There’s a good chance that it’s a race condition, though I have not looked at the code as yet.

Note that it is both a STANDALONE and mirroring registry.

Daemon

On the host(s) which access the local private registry, the docker daemon needs to be configured in order to allow access to the private repository. You can either set an environment variable, a command-line argument, or edit a config file. More information may be found in the Docker Documentation. I’ve used the config file; on debian based systems it’s generally located at /etc/default/docker. Add a line similar to the following at the end of the file:

Once done, the daemon will need to be restarted. On a debian system it is typically sudo service docker restart.

Note: In my test, I added the private registry to a host which already had port 80 bound. Hence the specification of the port. Your name, etc., will vary. I have written some thoughts about the pros and cons of various private registry schemes at Good Practices for Configuring Docker Private Registries.

Catching My Breath

At this point, the Raspberry Pi Swarm has:

1. Swarm
2. Consul and Registrator
3. A Private Registry and a Mirror
4. Monitoring is available through Consul and Registrator. I am not sure how well they work with ephemeral containers however. That is the subject of a future test. I may need to hack registrator to ignore ephemeral containers.

The remainder:

1. Bootstrapping the Swarm and basic services
2. Storage. I’ve got NFS working (it’s easy). I intend to evaluate:
   a. S3
   b. HDFS
   c. Ceph or Gluster
3. Log aggregation
4. Solving some “real” problems. ${WORK} is involved in Cheminformatics and authoritative chemical information. I’ve decided as a way to stretch the abilities of the swarm to do some substructure searching of chemical substances. I am not a chemist; I remember a good deal of my Advanced Placement Chemistry from ’87, but let’s just say I’m learning a lot. It’s good though, I think. I don’t know what’s impossible!

At this point I’d like to introduce Agni. Agni is my answer to Apache’s Lambda. However, it differs in two major areas:

• It’s Open Source and built upon Open Source.
• It supports multiple languages — not just Javascript.

On a high level, code is registered with Agni and a container image is created. A part of the creation process entails specifying an event/message with will trigger the running of an instance of the container. When events are recieved, a listener spawns instances via swarm, passing the details of the message to the newly created Docker container.

More on Agni shortly…. Meanwhile I’m back to the cluster and seeing how I can leverage the work which the good people at Hypriot have done with Docker Machine (and to a lesser degree Kitematic since I don’t have a Mac).