Dec 19

From the Trenches: Managing Organizational Change

Over the course of the past year I have been in a position to observe and take part in an organization which is undergoing changes on many levels including leadership, focus, and organizational structure. I’ve been involved as both an employee as well as in a team to study and report on organizational models, making recommendations regarding re-organizing an IT organization comprised of approximately four-hundred individuals.

Once an organization reaches a certain size it becomes difficult to overcome momentum in order to pivot, change, or go in a new direction. The longer an organization moves in a direction, the more ingrained patterns of behaviour become. In time it becomes like an old dirt road; enough traffic results in ruts. It might seem easier to just go with the flow and follow the ruts but in the long run the ruts become deeper and it becomes even harder to change when necessary.

Change is necessary — an organism which does not change is stagnant. However there are good ways to approach change and, conversely, bad ways. Here are some of my observations and understanding of what to do and what not to do in order to foster an environment for change.

Good Idea Bad Idea
Communicating the need for change and having town halls as an open forum for communication. Bringing in a motivational speaker who told the tale of how he freaked out one day and fired everyone at his business.
Establishing timelines for actions. Missing dates and milestones.
Openness where appropriate. Bringing in high-powered consultants (“Bobs”) who proceed to rule by fiat.
Be expeditious. Having the ‘Sword of Damocles’ dangling for months at a time.
Leadership communicates strategic direction. Communication disconnects result in tactical marching orders not given to worker bees.
Manage expectations. Loss of narrative. Gossip travels faster than the speed of light and beliefs, once established are almost impossible to change.
Be understanding of culture. Adopting a “one size fits all” mentality and dismissing concerns.
Sunlight in vendor management. Consultants should not only advocate for a single vendor to solve all problems (other than cost). This leads to questions of “kickbacks” or other improprieties. If the consultants are getting a kickback, that should be communicated. If they are not that also needs to be communicated.
Respecting employees Believing that people are infinitely fungible.
Having compassion. A surgeon makes cuts which are necessary in order to bring healing to the patient as a whole. Blindly cutting.

At the end of the day, I believe that change is necessary. How one goes about change drastically affects the outcome. Change is seldom fun but it can be good.

Dec 19

Nifty Things for Week of December 19

These are various and sundry interesting things I’ve come across during the week.

Random

  • The Icelandic word for raccoon is þvottabjörn which means “laundry bear.” — @SamuelMoen

All the Pretty Things

Cloud Computing

Data & Visualization

Programming and Development

Philosophy

Tools

  • Astral — tag and organize starred Github repositories
  • Big Picture — (almost) infinitely scrollable canvas for note taking and mind maps

Conferences

Dec 18

Avoid Screaming Customer Management Protocol and Get More Sleep

Screaming customers are the path to the dark side. Screaming
customers lead to phone calls. Phone calls lead to upset managers.
Upset managers lead to insomnia. — Yoda

A recent survey from Enterprise Management Associates identified user phone calls, also known as Screaming Customer Management Protocol (SCMP), as the most common way — by a large margin — that IT learns about application problems.

It doesn’t have to be that way.

Recent developments in DevOps practices as well as cloud technologies enable and encourage building monitoring and metric gathering from the ground floor. Just like pair programming, TDD, and agile methodologies help produce better software, DevOps practices such as monitoring and metrics enable better applications.

===================================================

Slides are available at http://aetherical.github.io/avoid-scmp-and-get-more-sleep/.

Dec 18

Using Docker Registry with Riak-CS

Intro

As a proof of concept, this uses the docker hectcastro/riak-cs container to create the riak-cs cluster. Some flavour of linux with a recent Docker version is assumed.

Riak-CS

Pull the github repository; it has a Makefile and scripts to bring up the cluster and test it. It can set up a haproxy container in front in order to proxy/loadbalance the cluster. Out of the box, it starts the docker instances on the same host. It wouldn’t be too difficult to use fleet or some other method to start the containers on multiple hosts.

I’ve found the following environment settings to be helpful for starting the riak cluster:

Within the docker-riak-cs directory, make start-cluster.

If you have not done a docker pull on the images; it might take a little while to run, depending on network speed.

Another quirk I’ve noticed is that when installing on a notebook, logging into multiple networks and/or vpn sessions, the IP stack can become a little confused. Rebooting helps; I expect there is a better way but I have not explored it.

Eventually, it will finish.

Once finished, the cluster should be tested — make test-cluster works well for this.

The registry needs a bucket created. s3cmd works well for this. In order to communicate with the cluster, you will need the admin-key and admin-secret. The following script can assist in querying the cluster for them:

You’ll need these; please make note of them.

Once you have these settings, invoke s3cmd --configure — it will ask a few questions and then test the settings. The parameters of interest are:

  • Access Key — the same as the ADMIN_KEY
  • Secret Key — the same as the ADMIN_SECRET
  • HTTP Proxy server name — use localhost
  • HTTP Proxy server port — Either use the PORT from above, or if you want to use haproxy, 8080.

Choose the default values for everything else, except that you wish to save the settings; choose Y for that.

Then add the bucket: s3cmd mb s3://BUCKET. Remember what you used for BUCKET.

Docker Repository

The docker repository can either be run as a container or at the host level.

Setup

It is advantageous to clone the git repository.

Having done so, the config/config_sample.yml provides a good base for the configuration. Add the following lines to the end of the ceph-s3 portion of the configuration:

Host Level

To run it at the host level requires pip. Install it with pip install docker-repository.

Assuming that you’ve run the previous commands, the following will start the registry on port 5000 (fix the path to docker-registry):

Docker Level

Pull the docker container — docker pull registry.

Replace $PATH_TO_DOCKER_REGISTRY below with the absolute path of the docker-registry git repository.

Check it out

The registry should now be running. Check the status with curl http://localhost:5000/v1/_ping.

Going Further

It is advantageous to do the following:

  1. Map registry to port 80 — this either requires root (running outside of docker) or the docker container. The advantage of doing so lies in not requiring the port when pushing or pulling from the registry — instead of docker pull registry:5000/my-container you can docker pull registry/my-container
  2. Set up mirroring in the docker registry — this allows for a local cache of docker containers so that they don’t need to be pulled down from the canonical docker registry.

Nov 19

Twelve Lessons Learned from a Startup Weekend

2014-11-16 20.41.59

This past weekend I participated in my first Startup Weekend Columbus.
During the course of which we built Fairflo, a marketplace for fair-trade items. The front end can use some work, but at the end of the weekend we had a fully functioning e-commerce site. I’ll write about the infrastructure separately, but as an aside Saturday night an attack from China was detected and defeated.

In no particular order, here are some lessons learned.

Lesson 1: Remember to take care of one’s self

Startup weekend is incredibly intense. There were approximately one hundred
extremely creative people all in a relatively small space — enough space in which to work, but close enough that creative juices were flowing amongst the groups. It’s very easy to find oneself living off the energy and forgetting to eat or otherwise feed the body. I’ve seen it elsewhere at festivals and conventions. In some senses it was more intense this time — perhaps due, in part to the close proximity; at conventions there are usually panels or other events which help to disrupt at least some of the flow. The organizers were really good about making sure that there was food available; as the weekend progressed more and more energy bars were placed out for the participants.

Sleep and/or other forms of downtime are needed as well.

Lesson 2: Practice the Pitch. Practice, practice, practice!

Given the limited amount of time to present the pitch, timing is very important, particularly if more than one person is presenting. Setting aside a couple of hours to practice and refine the pitch is very useful, especially for polishing and showmanship. Getting up in front of a crowd and talking is far easier than having a concise and succinct message. Providing a block of time is essential for being able to revise, refine, and review. Additionally the pitch should be given, if possible, to someone not on your team — perhaps a “trade” where teams can practice in front of each other.

Lesson 3: Mental Stealth Checks

Given the timeframe and pressure, it’s important for groups to periodically touch base with each other. In some senses it’s like a standup for an Agile team, but given rather more frequently. In addition to what people are working on and blockers, checking the mental/emotional state is very important. I think that a 10-15 minute standup including touching base/emotional check-in of, say, four times a day is probably about right:

  1. First thing in the morning
  2. Lunch time
  3. Dinner
  4. End of the day

It might be helpful if people are getting frustrated/frazzled to have them more often — even just a quick stretch and “Do you need anything” every couple of hours. Any more than that and flow is likely to be disrupted.

The organizers had daily meetups for all of us to touch base and talk about progress and/or needs. I’ll admit I felt a touch of resentment toward the meetups of all the teams — it was eating up time which I felt could have been better used for working with my team. I understand why they had the meetings; it is far too easy to fall down the rabbit hole and sometimes a break and/or change of pace is needed. In the moment, I felt as though it was disruptive of the gestalt which we were building.

Another reason for having these team meetings is to help maintain focus and keep on task — it is far too easy for Creatives to suffer from “Oooh, shiny!” and get distracted or to zoom from one thing to another when under pressure.

Lesson 4: Do high level planning Friday night

A quick high-level plan and/or breakdown of tasks and/or priorities are very helpful. Some things, are needed no matter what the problem domain:

  1. Overview of the problem space
  2. Feasibility and/or Customer studies
  3. Basic IT considerations (domain, email, etc.)

As such the sooner they can be worked on — or have an idea who is working upon them the better. Process can be important — not to be burdened by process, but it can provide a framework or track so that everyone is headed in the right direction.

Lesson 5: Build out the infrastructure Friday night

Leave as much time as possible for front-end details. Given the short time to pitch, having a “wow” helps translate to a “win”.

Lesson 6: It can be helpful to have a place to work together offsite

Since the venue closes each night, it can be helpful, but also a potential snare, to have a place in which to continue working. The snare lies in staying up too late!

Lesson 7: Have an afterparty/post mortem/closure

Startup weekend is zoom-zoom-zoom high energy build gestalt then stop. The let-down afterward can be really hard. Having space and time afterward provides closure which is really, really important. It doesn’t mean that the project is over, but it’s a time to come together and release some of the energy which has been building all weekend long.

The release can be seen in a lot of rituals, whether a benediction or releasing the quarters. Congregants come together for a purpose, fulfill the purpose and then have a closure. It’s something I think is needed.

Lesson 8: Thank the Organizers, Mentors, and Judges

Important but easy to forget. They have put a tremendous amount of effort into creating a space in which creative magic can happen.

Lesson 9: Think like a Boy Scout

Be Prepared — Boy Scout Motto

At any conference or event such as Startup Weekend there will always be a need for extra outlets, connectors, etc. Simply having a small power strip is a great way to make friends and meet new people. Bring one; you’ll be glad you did.

On the more expensive end of things, a micro-projector or access to a large monitor is useful to have people looking at the same thing without needing to huddle around a laptop. Also, it’s good for practicing presentations.

Lesson 10: Be Excellent to Each Other

A lot of what I’ve learned this last weekend has to do with people and small group dynamics. I was fortunate to be a part of a great group of people.

I discovered, too, that over the years I’ve picked up a lot more than I realized about groups, process, and business. I don’t know much about marketing or making a business plan, but I realize that I have learned a lot about people and how they work. I find that I have a better handle on project management than I might have thought, too.

In the course of Startup Weekend, there will likely be some conflict. A certain level of conflict can be good — if nothing else a better product can emerge through synthesis and refinement. However, it’s still imperative to be committed to each other and to the team.

Many cultures have a version of “Do unto others as you would have them do unto you” — in some ways I think that one of the most profound ways of expressing it is espoused by the most excellent Bill & Ted. On the surface they might appear foolish, but:

The Fool is the spirit in search of experience. He represents the mystical cleverness bereft of reason within us, the childlike ability to tune into the inner workings of the world. The sun shining behind him represents the divine nature of the Fool’s wisdom and exuberance, holy madness or ‘crazy wisdom’. On his back are all the possessions he might need. In his hand there is a flower, showing his appreciation of beauty. He is frequently accompanied by a dog, sometimes seen as his animal desires, sometimes as the call of the “real world”, nipping at his heels and distracting him. He is seemingly oblivious that he is walking toward a precipice, apparently about to step off. — The Fool

Lesson 11: Be Pragmatic

The perfect is the enemy of the good. — Voltaire

It is important during Startup Weekend to dream high. However, there are only 54 hours. There is only so much that a group of humans can achieve in that time. So it’s important to both strive for the best yet accept limitations.

Lesson 12: Have fun

It’s a lot of work. A lot of stress. But in order to be successful, it needs also be a lot of fun. If it’s not fun, you’re doing it wrong.


Thank you to everyone who made Startup Weekend possible. I’d like to thank the sponsors, organizers, mentors, judges, and other teams. But most of all I’d like to thank my teammates. You are all excellent!

Nov 14

Free Cloud Resources

This is a quick list of “free” cloud resources.

Oct 29

First do no harm

If you can, help others; if you cannot do that, at least do not harm them. — Dalai Lama

Over the past couple of years I’ve been trying to put into words my thoughts regarding the type of company where I’d like to work. There are companies I would not consider working for — their ethics and/or business model are radically different from what I consider good or right.

“Evil begins when you begin to treat people as things.” — Terry Pratchett, I Shall Wear Midnight (Discworld, #38)

  • I don’t want to work for a company which takes advantage of others.
  • I don’t want to work for predatory companies or ones which don’t treat their employees well.
  • I don’t want to go home at the end of the day feeling slimy or otherwise compromised.
  • I don’t want to work for a company which treats people as things.

I think technology really increased human ability. But technology cannot produce compassion. — Dalai Lama

I’ve had the fortune to work at a not-for-profit for the last nine years. Not being profit-driven means we’re not out to gouge people. Not that there’s anything wrong with seeking a profit, but seeking profit for the sake of profit leads to evil.

At the same time, without the relentless drive for profits it’s easier to focus on customers and service.

  • I don’t want to be associated with a company that views profits more valuable than people.

Don’t be evil. We believe strongly that in the long term, we will be better served — as shareholders and in all other ways — by a company that does good things for the world even if we forgo some short term gains. This is an important aspect of our culture and is broadly shared within the company. — Larry Page, Sergey Brin, Google IPO

A lot of people have scoffed at Google’s ideals in recent years. Nevertheless, it’s an admirable goal. I wish that more people or companies chose not to be evil in their day to day interactions with others.

The Agile Manifesto has brought a lot of change to Software Development. Other fields have recognized its inherent value and rewritten it to fit their circumstances.

I’d like to present a first draft of a manifesto for business.


A Business Manifesto

We are uncovering better ways of running a business and helping others do it.

Through this work we have come to value:

  • People and interactions over profits and prestige
  • Quality service over quantity of service
  • Customer relationships over contract negotiation
  • Flexibility over following a plan

That is, while there is value in the items on the right, we value the items on the left more.


In a nutshell I want to work for a company which values people — both inside and out of the company. I want to work where people strive to do things right.

Trust yourself. Create the kind of self that you will be happy to live with all your life. Make the most of yourself by fanning the tiny, inner sparks of possibility into flames of achievement. — Golda Meire

At the end of the day, I want to work for a company which isn’t evil. When I go home, I want to be able to look in the face of my daughter and not have to make excuses for the work that I do and the effect it has on others. And I want to be able to look myself in the mirror, too.

Be excellent to each other and Party On Dudes — Abraham Lincoln in Bill and Ted’s Excellent Adventure

Sep 28

Go executables are statically linked, except when they are not.

Generally GO executables are advertised as statically linked.

And they are, mostly, except for those times where they aren’t.

This is about one of those times, and what I discovered in the process.

Read the rest of this entry »

Sep 16

Docker Volumes Quirk

I discovered something interesting today regarding Docker and volumes — there’s a problem below with the call to run the private docker registry container:

docker run -d -v /opt/registry-cache/:/tmp/registry -p 5000:5000 registry

If you run that and curl -s http://localhost:5000, nothing is returned.

As it turns out, the trailing slash in /opt/registry-cache/ causes issues — the web proxy starts up, but the actual registry doesn’t run. In order to get it to work, the following needs to be done:

docker run -d -v /opt/registry-cache:/tmp/registry -p 5000:5000 registry

It’s amazing the difference a single character can make. Remove the extra '/' and it works as expected.

Jul 27

Using Openstack/Devstack Floating IPs from outside

I hope that this is of use to someone. After many hours of tearing down, building up, programming routers, etc., I’ve figured out why my devstack wasn’t allowing access from floating ips. I knew that the address was resolving (arp -a); I just couldn’t ssh to it (or do anything else with it)

It was the security rules.

So, I went in to the security rules in horizon and added the following rules to the default:

  1. Allow all CIDR ICMP traffic
  2. Allow all CIDR TCP traffic
  3. Allow all CIDR UDP traffic

Generally I wouldn’t advise opening it up totally — I’d open up applicable ports.  However, I’m happy to have it working!

Older posts «