Intro
As a proof of concept, this uses the docker hectcastro/riak-cs container to create the riak-cs cluster. Some flavour of linux with a recent Docker version is assumed.
Riak-CS
Pull the github repository; it has a Makefile and scripts to bring up the cluster and test it. It can set up a haproxy container in front in order to proxy/loadbalance the cluster. Out of the box, it starts the docker instances on the same host. It wouldn’t be too difficult to use fleet or some other method to start the containers on multiple hosts.
I’ve found the following environment settings to be helpful for starting the riak cluster:
|
1 2 3 4 5 6 7 8 |
export DOCKER_RIAK_CS_DEBUG=1 # turn on debug export DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING=1 # set up clustering export DOCKER_RIAK_CS_HAPROXY=1 # create the HAPROXY in # front of the cluster export DOCKER_RIAK_CS_CLUSTER_SIZE=3 # set the cluster size; generally # 5 is considered a minimum, but # for testing, three will suffice |
Within the docker-riak-cs directory, make start-cluster.
If you have not done a docker pull on the images; it might take a little while to run, depending on network speed.
Another quirk I’ve noticed is that when installing on a notebook, logging into multiple networks and/or vpn sessions, the IP stack can become a little confused. Rebooting helps; I expect there is a better way but I have not explored it.
Eventually, it will finish.
Once finished, the cluster should be tested — make test-cluster works well for this.
The registry needs a bucket created. s3cmd works well for this. In order to communicate with the cluster, you will need the admin-key and admin-secret. The following script can assist in querying the cluster for them:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
IP=`docker inspect riak-cs01|egrep IPAddress|sed -e 's/,//' -e 's/.*: //' -e 's/"//g'` if [[ ! -f insecure_key ]]; then curl -o insecure_key -fSL https://github.com/phusion/baseimage-docker/raw/master/image/insecure_key chmod 600 insecure_key fi ssh -i insecure_key root@$IP egrep "admin_key" /etc/riak-cs/app.config | cut -d'"' -f2 > admin_key ssh -i insecure_key root@$IP egrep "admin_secret" /etc/riak-cs/app.config | cut -d'"' -f2 > admin_secret PORT=`docker port riak-cs01 8080 | cut -d":" -f2` echo IP=$IP echo PORT=$PORT echo ADMIN_KEY=`cat admin_key` echo ADMIN_SECRET=`cat admin_secret` |
You’ll need these; please make note of them.
Once you have these settings, invoke s3cmd --configure — it will ask a few questions and then test the settings. The parameters of interest are:
- Access Key — the same as the
ADMIN_KEY - Secret Key — the same as the
ADMIN_SECRET - HTTP Proxy server name — use
localhost - HTTP Proxy server port — Either use the
PORTfrom above, or if you want to use haproxy,8080.
Choose the default values for everything else, except that you wish to save the settings; choose Y for that.
Then add the bucket: s3cmd mb s3://BUCKET. Remember what you used for BUCKET.
Docker Repository
The docker repository can either be run as a container or at the host level.
Setup
It is advantageous to clone the git repository.
Having done so, the config/config_sample.yml provides a good base for the configuration. Add the following lines to the end of the ceph-s3 portion of the configuration:
|
1 2 3 |
boto_proxy: _env:BOTO_PROXY_HOST boto_proxy_port: _env:BOTO_PROXY_PORT |
Host Level
To run it at the host level requires pip. Install it with pip install docker-repository.
Assuming that you’ve run the previous commands, the following will start the registry on port 5000 (fix the path to docker-registry):
|
1 2 3 4 5 6 7 8 9 10 11 12 |
export DOCKER_REGISTRY_CONFIG=docker-registry/config/config.yml export AWS_KEY=`cat admin_key` export AWS_SECRET=`cat admin_secret` export AWS_BUCKET=lolrus export SETTINGS_FLAVOR=ceph-s3 export LOGLEVEL=debug export BOTO_PROXY_HOST=127.0.0.1 export BOTO_PROXY_PORT=$PORT export AWS_HOST=s3.amazonaws.com export AWS_PORT=80 docker-registry |
Docker Level
Pull the docker container — docker pull registry.
Replace $PATH_TO_DOCKER_REGISTRY below with the absolute path of the docker-registry git repository.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
docker run \ -v $PATH_TO_DOCKER_REGISTRY:/docker-registry \ -p 5000:5000 \ -e DOCKER_REGISTRY_CONFIG=/docker-registry/config/config.yml \ -e AWS_KEY=`cat admin_key` \ -e AWS_SECRET=`cat admin_secret` \ -e AWS_BUCKET=lolrus \ -e SETTINGS_FLAVOR=ceph-s3 \ -e LOGLEVEL=debug \ -e BOTO_PROXY_HOST=127.0.0.1 \ -e BOTO_PROXY_PORT=$PORT \ -e AWS_HOST=s3.amazonaws.com \ -e AWS_PORT=80 \ registry |
Check it out
The registry should now be running. Check the status with curl http://localhost:5000/v1/_ping.
Going Further
It is advantageous to do the following:
- Map registry to port 80 — this either requires root (running outside of docker) or the docker container. The advantage of doing so lies in not requiring the port when pushing or pulling from the registry — instead of
docker pull registry:5000/my-containeryou candocker pull registry/my-container - Set up mirroring in the docker registry — this allows for a local cache of docker containers so that they don’t need to be pulled down from the canonical docker registry.
2 comments
Miguel Marques
June 26, 2015 at 4:54 am (UTC -5) Link to this comment
Hi, thanks for the article. I have been playing with that docker image myself.
How do you deal with container data persistence? And the config files for riak, riak-cs and stanchion use the IP address of the docker container, as we can see in bin/stanshion.sh bin/riak.sh bin/riak-cs.sh “IP_ADDRESS=$(ip -o -4 addr list eth0 | awk ‘{print $4}’ | cut -d/ -f1)” this IP Address is replacing 127.0.0.1 in the config files for those 3 components. If the docker container ip changes it is hard to make it work again, even updating the ip in the config files, and following this solution: http://stackoverflow.com/a/13863702/249687.
I wonder if it works if we map all the important port to the same ones in the host and use the host ip address instead, I will test that.
Matt Williams
August 14, 2015 at 11:57 pm (UTC -5) Link to this comment
You will need to create a data volume, or mount the position on the filesystem with
-v. Yes, the host IP should work.