Swarming Raspberry Pi: Docker Swarm Discovery Options

Docker Swarm supports a variety of methods for discovering swarm members. Each has arguments in its favor. In this post I shall discuss the various methods and thoughts regarding each.

Background

I originally started with the idea of having a portable cluster, a “cloud in a box” if you will, so that I could go and give talks without having to worry about network dependencies and so forth. I also was intrigued by the idea of low power, inexpensive devices which could be used for building a cloud. Two days after my initial purchase of 5 Pi B+, the Pi 2 was released. Despite my initial grump, I realized that this presented possibilities for distributing workloads across a heterogenous environment which is an interesting problem space — determining how best to distribute work across an infrastructure.

I still have the goal, for the present of having a portable cloud. I’ve been challenged, however, to build a larger one than Britain’s GCHQ Pi cloud. It is tempting. Since they’re using all single core Pi’s, it wouldn’t be terribly difficult to build a cloud with more oomph and far fewer nodes. Of course, if the workload is IO intensive then more members are needed.

At present, my cloud is consisting of the following:

1. 5x Pi B+ Worker Nodes, 16GB micro SD
2. 5x Pi 2B Worker Nodes, 16GB micro SD
3. 1x Pi 2B Master Node (temporarily being used as a porting/compiling node), 16GB Micro SD
4. 2x Pi 2B Data Nodes (one of these will become a docker registry, among other things)
   a. One has 2x 240GB SSD
   b. One has a 240GB SSD and a 160GB Spinning disk (for “slow” data)
5. 16 Port 100Mbit Switch. This may shortly be swapped out for gigabit switch(es).

Criteria for Evaluation

I strongly believe that metrics, monitoring, and alerting are necessary in building any infrastructure.

I am seeking maximum portability; my Cloud in a Box™ should be able to do Real Work™ without depending on anything outside the cluster. Additionally, the less I need to know ahead of time the better. Names trump numbers — if I can use a name or use a name to look up a number that is better than having to remember a number.

Given the limited resources of the Pi, lightweight solutions are preferred over heavyweight, saving where they can serve dual purposes.

The Contendors

The list of Discovery Services can be found in the Docker Documentation.

Hosted Discovery Service

The hosted discovery service presents an easy way to test and get started with Swarm. Swarm communicates with the Docker Hub in order to maintain a list of swarm members.

The Good

It’s easy, presented in the tutorial, and is supported by Docker.

The Bad

Unfortunately the requirement of connecting the the Docker Hub means that it’s not self contained; in order for it to work a network connection is needed.

As of today, there a couple of issues with it:

1. There is no way to remove a host from the swarm.

2. docker -H $SWARM_MANAGER info returns what I believe is an incorrect count:

As an example, in the case of apis-rpi-04, it’s claiming that there are six (6) containers. However, there are not six (6) containers running:

There are, however, six in the ps -a results:

On a whim, I removed the containers which were not up:

At this point, info returns the three (3) which I’d expect. However, on further investigation, it turns out that docker info outside of swarm returns the total number of containers, not the number of running containers. I’ve
opened an issue about this. I think that having an entry for the number of containers running would be useful, but barring that documentation is good.

Static File Describing the Cluster

In this case, a file is used by all the members which has a list of all the hosts IP addresses and ports.

The Good

It’s pretty simple.

The Bad

Since my cluster is portable, I don’t necessarily know what the IP addresses are — I may happen to be on a network where the addresses I’ve chosen are already in use. For simplicity sake I don’t want to have to worry at this point about NAT translation.

The file needs to be copied to all of the servers. Additionally, it violates principles espoused by The Twelve-Factor App — primarily there is a configuration artifact which needs to be maintained.

A Static List of IPs

The Good

Same as the file list, but this also has the added goodness of being more 12 Factor compliant.

The Bad

Same as the file list.

Range pattern for IP addresses

The Good

The Bad

etcd

When I investigated coreos/etcd on the Raspberry Pi wouldn’t compile without patching the code — to whit, a structure needs to be edited. I don’t view it as very portable and I have concerns whether the structure change will keep it from working properly with other frameworks. At least for the moment I don’t consider it to be a good choice.

zookeeper

The Good

Established codebase with lots of users.

The Bad

It’s really heavyweight compared to some of the other options. However, if plannng to use hadoop or anthing in the hadoop ecosystem, it might be a good choice.

Consul

The Good

Consul can serve multiple purposes — service discovery, monitoring, and DNS. Additionally there’s a fairly useful web ui which provides a dashboard showing the status of the members.

It’s fairly lightweight — the client takes approximately 3MB:

The Server, on the other hand, takes about 12MB for managing 13 hosts:

That 821 minutes is over 11 days:

That’s about 1.25 hours/day of CPU time. By comparison, the client is a bit lighter:

The Bad

It’s rather chattier than the other methods. Every 10 seconds or so the client wakes up and does some work.

Verdict

For the moment, based upon my goals and an analysis of the good and the bad of the various methods available today, I think that consul is the best choice at this point.