«

»

Mar 21

Swarming Raspberry Pi – Part 1

Here’s the first in a series of building a robust docker swarm of raspberry pi hosts. This installment goes through installing the Pi(s), modifiying the Docker Daemon config, and starting up a test to verify that everyone can join the cluster.

The cluster which I am using consists of

  • Raspberry Pi B+ x 5
  • Raspberry Pi 2 B x 5

In order to verify that the cluster is working properly, there should be ten (10) members, five (5) of which have 4 cores.

Obtaining the OS and Setup

The wonderful folks at Hypriot have produced a raspberry image based on Raspbian, but with Docker 1.5.0. The image and instructions can be found at Heavily ARMed after major upgrade: Raspberry Pi with Docker 1.5.0 – Docker Pirates ARMed with explosive stuff.

Note: The image resizes itself to fit the card upon the first boot and then reboots. This takes a little bit (minutes at most). Just don’t be surprised when it reboots.

Hostnames

In order for this to work properly, the hosts need a unique name. It’s sometimes possible to configure DHCP to do this, but I went ahead and edited /etc/hostname myself since I wanted the most portability possible.

Like all good sysadmins, I am lazy. Doing the same thing over and over is boring. Besides, that’s what computers are for. So I wrote a script to make copies of the image and then edit the hostnames. Once that’s done I can then proceed to write the images to the SD cards.

The following script takes one argument, the location of the image you wish to modify — this should work for more than just Hypriot; it should work for just about any Rasberry Pi image. At present, it creates 10 instances and the NAME starts with apis-rpi- — you will likely want to change those. I may make the file a bit more generic at some point.

The Pi instances all need network connectivity which can be seen by the swarm manager; a switch works well. You can either physically assign IP addresses or use DHCP.

Updating Docker

In order for swarm instances to communicate, the docker daemon needs to be bound to a TCP port.

  1. sudo service docker stop
  2. Edit /etc/default/docker (vi is installed on the image) and replace DOCKER_OPTS="--storage-driver=overlay -D" with DOCKER_OPTS="--storage-driver=overlay -D -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock"
  3. sudo service docker restart

Docker should be running once again. I’ve left the unix socket enabled so that the “default” behaviour still works.

WARNING There are some security issues with this; a port is opened to anyone who can reach the host at which point they can start containers, etc.. Other than using this for play/testing on a private network, TLS needs to be configured.

To verify that the daemon is running with both the port and the socket, do docker info and docker -H 127.0.0.1:2375. You should see something like the following for both:

Obtaining Swarm

I’ve created a Raspberry Pi docker container for swarm. To install it, you can simply pull nimblestratus/rpi-swarm. With more than a couple Raspberry Pis, this can be a bit tedious. clusterssh makes this a lot easier — it allows you to type in the same commands to multiple servers at once.

Discovery Service

In future installments I’ll be getting a different discovery service running, but for this test the default works.

Starting the Nodes

On each node, execute the following: (this assumes that you’re using wired ethernet; for wireless you’ll need to change eth0 to most likely wlan0)

It can be helpful to verify that the agent has started with docker ps. If you don’t see it running, then there’s a problem somewhere.

Starting the Manager

I’m running the manager on my laptop; consequently I’m using the swarm container instead of nimblestratus/rpi-swarm:

Once this comes back, let’s check out the swarm:

Ok, let’s test…

You can find the url like this:

At this point there isn’t any service discovery; that’s for the next increment….

Pop the URL into a browser and you should see:

swarm-result

It's Alive!

It’s Alive!

Next time… setting up a discovery service…

10 comments

3 pings

Skip to comment form

  1. stefanscherer

    That rocks! I’ll try it with myself with my smaller cluster of 3xPi2 + 1xPi1.

    But I think we have to find an easier way to customize the hostname, perhaps a config file in the VFAT /boot partition…

  2. BOK

    This is great, even with only one RPi in use!
    Keep in mind that if you run from OS X and Boot2Docker, you have to execute the “docker -H” commands INSIDE the VM (enter with “boot2docker ssh”).

  3. Dieter Reuter

    Matt, thanks for sharing this cool tutorial.

    I just built my 6x node cluster (5x Pi 2, 1x Pi B+). With your blog post it was easy to setup.
    Now, we can clearly say, that HypriotOS is already supporting Docker Swarm!

    The best thing is, this cluster consumes 9.5 Watts only. For me that’s the way to go, using power-efficient clusters and stepping into “Real Green IT”.

    BTW: we’ll automate a few more steps, stay tuned…

  4. Matt Williams

    Glad to help! Thank you for all of your work as well; it made life much, much easier.

    For some of the automation, maybe something like cloud-init might help….

  5. Matt Williams

    It would make things a bit easier. I wonder if basing it on cloud-init might work…

  6. Matt Williams

    Makes sense — thanks for the tip.

  7. Bargepole

    This looks rather interesting.
    Must one use a different OS, for compatibility reasons I’d like to stick to good ols Debian
    Wheezy – how can one achieve this?

  8. Matt Williams

    Hypriot is actually wheezy, just stripped down a bit with support for Docker baked in. Stock Debian doesn’t have support for Docker without some serious hacking at which point you’re better off starting with Hypriot and then adding packages you want. To compare, the Hypriot image is about 1G whereas the Raspbian is 4G. Admittedly the Hypriot folk are building with an emphasis on a light image focused on running containers, but since they’re based on Debian it’s just a matter of ‘apt-get’ing the packages you think are missing.

  9. goliatone

    Thanks for this series, really helpful. Initially I wanted to use zettio/weave, the idea of having a meta network to which you can plug a device anytime and have it available is really interesting. Do you have any idea how weave would integrate into this swarm architecture you are putting together?

  10. Matt Williams

    I’m interested in weave, too. Unfortunately as of the last I looked (the last week or so), someone reported on the Hypriot site that weave wasn’t working. I don’t remember the exact details, but I think it had something to do with the Docker port. Consequently, I’ve not given it a whole lot of thought.

    I just did a little googling, however, and I see that there’s a number of folk who have used weave with the Pi. I’ll look at porting it this week and see what I get.

    As far as integration, I think that a lot of it would depend on the discovery service used. Static addresses or a file containing a list of members would not work very well. Consul, etcd, or the Docker Hub token would likely work. An IP range might work with weave. I think that if you had a way of finding the “manager” node and assuming it was in the weave network then it should all work if you required that all of the addresses were weave addresses.

    There is a possibility of bottlenecking, but that’s a potential issue with weave anyway.

  1. Swarming Raspberry Pi, Part 2: Registry & Mirror » Ramblings

    […] Previous: Part 1 […]

  2. 350+ ???????? ????????, ???? ? ???????????? ??? ?????? ? Docker — ? ???? ??

    […] Matthewkwilliams: ????????????? Swarm ??? Raspberry Pi; […]

  3. ?? Docker ???? - ???????

    […] Swarming Raspberry Pi – Part 1 […]

Leave a Reply

%d bloggers like this: