Dec 24
Merging Records with Uncertainty
I’m in the process of revisiting and migrating a service I’d first written almost five years ago; one of the backend data sources has changed (drastically!) so I’m looking at refactoring as well as streamlining in the process — parsing the new backend takes longer than the old. Part of the reason for doing so is related to (very rare, but still possible) timeouts of the feed with the old, faster version.
Background
I was asked to build a drop-in replacement for a legacy application which merged user information from a telecom system and ldap in order to create a number of feeds which were used by other systems, such as an employee locator, as well as feeding back into ldap so that the telecom system would be the canonical source for certain pieces of user data such as room number, name, and phone extension.
The original software was written in BPL (or PL/B?) and had originally been written decades prior. Other tools grew organically around it. The “owner” of the process retired and I was brought in to replace it.
The first iteration resulted in (mainly due to politics at the time) a jar file which embedded JRuby and a simple Ruby on Rails application. As such the server could be invoked via java -jar and the feeds were pulled via cron jobs. Quirky, but it worked. I’ve only had a couple of requests for help with it over that time — the first related to GIGO and the second requiring a reboot of the server (it had been up a very long time). Since it was originally written there have been vast improvements in terms of Java, JRuby, and Rails. I’m pretty surprised it was that rock solid.
Fast forward several years and there is a new telecom system. The old allowed ldap queries to access the data. The new, unfortunately, does not expose the data in this fashion. Rather, it requires a cumbersome and potentially brittle process of logging in via mechanize, walking through a few pages to first generate a CSV file and then download said file. Additionally there are schema differences between the old and new system.
Uncertainty
One of the feeds, for locating employees locations and/or phone numbers, merges information from two systems. However, there is not a shared primary key so the merger needs to happen based on the values of several fields which may not be in sync between the systems:
- First Name
- Last Name
- Department
- Extension
- Location
Given constraints when the first system was created, I settled on a relatively naive implementation which is O(n^2). It worked, however, which given the services were each called once daily was more than sufficient. (Premature optimization is the root of all evil!)
All of the telecom records are in one bucket. All of the ldap records are in another.
- The telecom record is compared to each ldap user record in turn, calculating a weight:
|
1 2 3 4 5 6 7 8 9 10 |
def match_weight(u) weight = 0 weight += 3 if u.first_name == @first_name.upcase weight += 1 if u.last_name == @last_name.upcase weight += 1 if u.department == @department.upcase weight += 1 if u.extension == @extension.upcase weight += 1 if u.location == @location.upcase return weight end |
- The highest weight and the index to that record is saved unless a threshold is reached. The assumption being that while it is possible for one or more attributes to change, it is unlikely that all of them are going to change at the same time with the first name being the least likely to change.
- The highest rated user record is merged with the telecom record and the matched record is deleted from the pool.
- Processing continues.
Worst case n + (n - 1) + (n -2) ... 1 comparisons are needed. The geek in me recognizes that this happens to have been made famous by Carl Gauss and can be expressed by n(n + 1)/2. It’s still O(n^2), however.
Refactoring
I think that a somewhat more sophisticated algorithm would provide O(n) behaviour at the expense of memory.
It occurs to me that by pre-calculating hashes of the values I can set up a series of bucket hashes for the various attributes I can then do a quick merge of the buckets to determine which record is the best match. Additionally Ruby allows me to easily keep the weighting of the first name:
|
1 2 3 |
1.9.3-p551 :001 > [1,2,3] * 3 => [1, 2, 3, 1, 2, 3, 1, 2, 3] |
Then determining the best match would then look something like this (assuming that User is defined elsewhere and has proper attributes):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
class Merger @attributes = (first_name last_name department extension location) # @buckets is a hash of hashes with a bucket hash # for each attribute; def fill_buckets(users) # Let ruby pre-initialize the buckets; we just worry about populating # This lets us do something like # @buckets["first_name"]["matt"] << id # id == 1 # @buckets then looks like: # {"first_name"=>{"matt"=>[1]}} @buckets = Hash.new {|h,k| h[k] = Hash.new {|h1,k1| h1[k1] = Array.new}} @users.each do |user| @attributes.each do |attr| weight = attr.equal?("first_name") ? 3 : 1 @buckets[attr][user.get_instance_variable[attr]] << ([user.id] * weight) end end end # match matches a user to the best match from the bucket def match(user) hits = @attributes.map do |attribute| @buckets[attribute][user.get_instance_variable(attribute)] end # hits will look something like this: # [[1, 2, 3, 1, 2, 3, 1, 2, 3], [1, 4, 5], [2, 3, 5], [1]] hits.flatten! # [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 4, 5, 2, 3, 5, 1] hits.sort! # [1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 5, 5] # once sorted, we can use Enumerable#chunk and # Enumerable#sort_by to determine which id has the # most hits. It's reversed so we have the element # with the greatest number of hits appearing first # in our array counts = hits.chunk {|id| id}.sort_by {|id,count| count.size}.reverse # [[1, [1, 1, 1, 1, 1]], [3, [3, 3, 3, 3]], [2, [2, 2, 2, 2]], [5, [5, 5]], [4, [4]]] match = counts.first.first end end |
Conclusion
As with anything else, there are trade-offs. While this code will run faster, it does so at the cost of memory footprint and complexity. The code is a bit more complex than the original version. However, in this case I think it is worth it.
It’s still not 100% perfect; however the case in which all of the attributes change is very, very unlikely. Moreover, without a shared primary key, there is no way that we can always be certain of a match. I’m pretty confident that the case in which all of the attributes change is so unlikely as to be non-existent, given the dataset.
Also, I’ll admit the alterior motive of having a good thought exercise for Christmas Eve. I learned something new about Ruby’s Enumerable. It’s a good day when I get to learn something new.
Dec 19
From the Trenches: Managing Organizational Change
Over the course of the past year I have been in a position to observe and take part in an organization which is undergoing changes on many levels including leadership, focus, and organizational structure. I’ve been involved as both an employee as well as in a team to study and report on organizational models, making recommendations regarding re-organizing an IT organization comprised of approximately four-hundred individuals.
Once an organization reaches a certain size it becomes difficult to overcome momentum in order to pivot, change, or go in a new direction. The longer an organization moves in a direction, the more ingrained patterns of behaviour become. In time it becomes like an old dirt road; enough traffic results in ruts. It might seem easier to just go with the flow and follow the ruts but in the long run the ruts become deeper and it becomes even harder to change when necessary.
Change is necessary — an organism which does not change is stagnant. However there are good ways to approach change and, conversely, bad ways. Here are some of my observations and understanding of what to do and what not to do in order to foster an environment for change.
| Good Idea | Bad Idea |
|---|---|
| Communicating the need for change and having town halls as an open forum for communication. | Bringing in a motivational speaker who told the tale of how he freaked out one day and fired everyone at his business. |
| Establishing timelines for actions. | Missing dates and milestones. |
| Openness where appropriate. | Bringing in high-powered consultants (“Bobs”) who proceed to rule by fiat. |
| Be expeditious. | Having the ‘Sword of Damocles’ dangling for months at a time. |
| Leadership communicates strategic direction. | Communication disconnects result in tactical marching orders not given to worker bees. |
| Manage expectations. | Loss of narrative. Gossip travels faster than the speed of light and beliefs, once established are almost impossible to change. |
| Be understanding of culture. | Adopting a “one size fits all” mentality and dismissing concerns. |
| Sunlight in vendor management. | Consultants should not only advocate for a single vendor to solve all problems (other than cost). This leads to questions of “kickbacks” or other improprieties. If the consultants are getting a kickback, that should be communicated. If they are not that also needs to be communicated. |
| Respecting employees | Believing that people are infinitely fungible. |
| Having compassion. A surgeon makes cuts which are necessary in order to bring healing to the patient as a whole. | Blindly cutting. |
At the end of the day, I believe that change is necessary. How one goes about change drastically affects the outcome. Change is seldom fun but it can be good.
Dec 19
Nifty Things for Week of December 19
These are various and sundry interesting things I’ve come across during the week.
Random
- The Icelandic word for raccoon is þvottabjörn which means “laundry bear.” — @SamuelMoen
All the Pretty Things
Cloud Computing
- StackStorm vs. AWS Lambda: Event-Driven Computing vs. Event-Driven Operations
- AWS Lambda
- Announcing Snappy Ubuntu Core
Data & Visualization
- Probabilistic Data Structures for Web Analytics and Data Mining
- Hokusai — Sketching Streams in Real Time
- Probabilistic Data Structures for Go
- MetricsGraphics.js
- Self Organizing Map
- Data Visualization with JavaScript
- Mermaid — Generation of diagram and flowchart from text in a similar manner as markdown
- Anode — Utility for analyzing graphite metrics
- Use Random Forest: Testing 179 Classifiers on 121 Datasets
Programming and Development
Philosophy
Tools
- Astral — tag and organize starred Github repositories
- Big Picture — (almost) infinitely scrollable canvas for note taking and mind maps
Conferences
Dec 18
Avoid Screaming Customer Management Protocol and Get More Sleep
Screaming customers are the path to the dark side. Screaming
customers lead to phone calls. Phone calls lead to upset managers.
Upset managers lead to insomnia. — Yoda
A recent survey from Enterprise Management Associates identified user phone calls, also known as Screaming Customer Management Protocol (SCMP), as the most common way — by a large margin — that IT learns about application problems.
It doesn’t have to be that way.
Recent developments in DevOps practices as well as cloud technologies enable and encourage building monitoring and metric gathering from the ground floor. Just like pair programming, TDD, and agile methodologies help produce better software, DevOps practices such as monitoring and metrics enable better applications.
===================================================
Slides are available at http://aetherical.github.io/avoid-scmp-and-get-more-sleep/.
Dec 18
Using Docker Registry with Riak-CS
Intro
As a proof of concept, this uses the docker hectcastro/riak-cs container to create the riak-cs cluster. Some flavour of linux with a recent Docker version is assumed.
Riak-CS
Pull the github repository; it has a Makefile and scripts to bring up the cluster and test it. It can set up a haproxy container in front in order to proxy/loadbalance the cluster. Out of the box, it starts the docker instances on the same host. It wouldn’t be too difficult to use fleet or some other method to start the containers on multiple hosts.
I’ve found the following environment settings to be helpful for starting the riak cluster:
|
1 2 3 4 5 6 7 8 |
export DOCKER_RIAK_CS_DEBUG=1 # turn on debug export DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING=1 # set up clustering export DOCKER_RIAK_CS_HAPROXY=1 # create the HAPROXY in # front of the cluster export DOCKER_RIAK_CS_CLUSTER_SIZE=3 # set the cluster size; generally # 5 is considered a minimum, but # for testing, three will suffice |
Within the docker-riak-cs directory, make start-cluster.
If you have not done a docker pull on the images; it might take a little while to run, depending on network speed.
Another quirk I’ve noticed is that when installing on a notebook, logging into multiple networks and/or vpn sessions, the IP stack can become a little confused. Rebooting helps; I expect there is a better way but I have not explored it.
Eventually, it will finish.
Once finished, the cluster should be tested — make test-cluster works well for this.
The registry needs a bucket created. s3cmd works well for this. In order to communicate with the cluster, you will need the admin-key and admin-secret. The following script can assist in querying the cluster for them:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
IP=`docker inspect riak-cs01|egrep IPAddress|sed -e 's/,//' -e 's/.*: //' -e 's/"//g'` if [[ ! -f insecure_key ]]; then curl -o insecure_key -fSL https://github.com/phusion/baseimage-docker/raw/master/image/insecure_key chmod 600 insecure_key fi ssh -i insecure_key root@$IP egrep "admin_key" /etc/riak-cs/app.config | cut -d'"' -f2 > admin_key ssh -i insecure_key root@$IP egrep "admin_secret" /etc/riak-cs/app.config | cut -d'"' -f2 > admin_secret PORT=`docker port riak-cs01 8080 | cut -d":" -f2` echo IP=$IP echo PORT=$PORT echo ADMIN_KEY=`cat admin_key` echo ADMIN_SECRET=`cat admin_secret` |
You’ll need these; please make note of them.
Once you have these settings, invoke s3cmd --configure — it will ask a few questions and then test the settings. The parameters of interest are:
- Access Key — the same as the
ADMIN_KEY - Secret Key — the same as the
ADMIN_SECRET - HTTP Proxy server name — use
localhost - HTTP Proxy server port — Either use the
PORTfrom above, or if you want to use haproxy,8080.
Choose the default values for everything else, except that you wish to save the settings; choose Y for that.
Then add the bucket: s3cmd mb s3://BUCKET. Remember what you used for BUCKET.
Docker Repository
The docker repository can either be run as a container or at the host level.
Setup
It is advantageous to clone the git repository.
Having done so, the config/config_sample.yml provides a good base for the configuration. Add the following lines to the end of the ceph-s3 portion of the configuration:
|
1 2 3 |
boto_proxy: _env:BOTO_PROXY_HOST boto_proxy_port: _env:BOTO_PROXY_PORT |
Host Level
To run it at the host level requires pip. Install it with pip install docker-repository.
Assuming that you’ve run the previous commands, the following will start the registry on port 5000 (fix the path to docker-registry):
|
1 2 3 4 5 6 7 8 9 10 11 12 |
export DOCKER_REGISTRY_CONFIG=docker-registry/config/config.yml export AWS_KEY=`cat admin_key` export AWS_SECRET=`cat admin_secret` export AWS_BUCKET=lolrus export SETTINGS_FLAVOR=ceph-s3 export LOGLEVEL=debug export BOTO_PROXY_HOST=127.0.0.1 export BOTO_PROXY_PORT=$PORT export AWS_HOST=s3.amazonaws.com export AWS_PORT=80 docker-registry |
Docker Level
Pull the docker container — docker pull registry.
Replace $PATH_TO_DOCKER_REGISTRY below with the absolute path of the docker-registry git repository.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
docker run \ -v $PATH_TO_DOCKER_REGISTRY:/docker-registry \ -p 5000:5000 \ -e DOCKER_REGISTRY_CONFIG=/docker-registry/config/config.yml \ -e AWS_KEY=`cat admin_key` \ -e AWS_SECRET=`cat admin_secret` \ -e AWS_BUCKET=lolrus \ -e SETTINGS_FLAVOR=ceph-s3 \ -e LOGLEVEL=debug \ -e BOTO_PROXY_HOST=127.0.0.1 \ -e BOTO_PROXY_PORT=$PORT \ -e AWS_HOST=s3.amazonaws.com \ -e AWS_PORT=80 \ registry |
Check it out
The registry should now be running. Check the status with curl http://localhost:5000/v1/_ping.
Going Further
It is advantageous to do the following:
- Map registry to port 80 — this either requires root (running outside of docker) or the docker container. The advantage of doing so lies in not requiring the port when pushing or pulling from the registry — instead of
docker pull registry:5000/my-containeryou candocker pull registry/my-container - Set up mirroring in the docker registry — this allows for a local cache of docker containers so that they don’t need to be pulled down from the canonical docker registry.
Nov 19
Twelve Lessons Learned from a Startup Weekend
This past weekend I participated in my first Startup Weekend Columbus.
During the course of which we built Fairflo, a marketplace for fair-trade items. The front end can use some work, but at the end of the weekend we had a fully functioning e-commerce site. I’ll write about the infrastructure separately, but as an aside Saturday night an attack from China was detected and defeated.
In no particular order, here are some lessons learned.
Lesson 1: Remember to take care of one’s self
Startup weekend is incredibly intense. There were approximately one hundred
extremely creative people all in a relatively small space — enough space in which to work, but close enough that creative juices were flowing amongst the groups. It’s very easy to find oneself living off the energy and forgetting to eat or otherwise feed the body. I’ve seen it elsewhere at festivals and conventions. In some senses it was more intense this time — perhaps due, in part to the close proximity; at conventions there are usually panels or other events which help to disrupt at least some of the flow. The organizers were really good about making sure that there was food available; as the weekend progressed more and more energy bars were placed out for the participants.
Sleep and/or other forms of downtime are needed as well.
Lesson 2: Practice the Pitch. Practice, practice, practice!
Given the limited amount of time to present the pitch, timing is very important, particularly if more than one person is presenting. Setting aside a couple of hours to practice and refine the pitch is very useful, especially for polishing and showmanship. Getting up in front of a crowd and talking is far easier than having a concise and succinct message. Providing a block of time is essential for being able to revise, refine, and review. Additionally the pitch should be given, if possible, to someone not on your team — perhaps a “trade” where teams can practice in front of each other.
Lesson 3: Mental Stealth Checks
Given the timeframe and pressure, it’s important for groups to periodically touch base with each other. In some senses it’s like a standup for an Agile team, but given rather more frequently. In addition to what people are working on and blockers, checking the mental/emotional state is very important. I think that a 10-15 minute standup including touching base/emotional check-in of, say, four times a day is probably about right:
- First thing in the morning
- Lunch time
- Dinner
- End of the day
It might be helpful if people are getting frustrated/frazzled to have them more often — even just a quick stretch and “Do you need anything” every couple of hours. Any more than that and flow is likely to be disrupted.
The organizers had daily meetups for all of us to touch base and talk about progress and/or needs. I’ll admit I felt a touch of resentment toward the meetups of all the teams — it was eating up time which I felt could have been better used for working with my team. I understand why they had the meetings; it is far too easy to fall down the rabbit hole and sometimes a break and/or change of pace is needed. In the moment, I felt as though it was disruptive of the gestalt which we were building.
Another reason for having these team meetings is to help maintain focus and keep on task — it is far too easy for Creatives to suffer from “Oooh, shiny!” and get distracted or to zoom from one thing to another when under pressure.
Lesson 4: Do high level planning Friday night
A quick high-level plan and/or breakdown of tasks and/or priorities are very helpful. Some things, are needed no matter what the problem domain:
- Overview of the problem space
- Feasibility and/or Customer studies
- Basic IT considerations (domain, email, etc.)
As such the sooner they can be worked on — or have an idea who is working upon them the better. Process can be important — not to be burdened by process, but it can provide a framework or track so that everyone is headed in the right direction.
Lesson 5: Build out the infrastructure Friday night
Leave as much time as possible for front-end details. Given the short time to pitch, having a “wow” helps translate to a “win”.
Lesson 6: It can be helpful to have a place to work together offsite
Since the venue closes each night, it can be helpful, but also a potential snare, to have a place in which to continue working. The snare lies in staying up too late!
Lesson 7: Have an afterparty/post mortem/closure
Startup weekend is zoom-zoom-zoom high energy build gestalt then stop. The let-down afterward can be really hard. Having space and time afterward provides closure which is really, really important. It doesn’t mean that the project is over, but it’s a time to come together and release some of the energy which has been building all weekend long.
The release can be seen in a lot of rituals, whether a benediction or releasing the quarters. Congregants come together for a purpose, fulfill the purpose and then have a closure. It’s something I think is needed.
Lesson 8: Thank the Organizers, Mentors, and Judges
Important but easy to forget. They have put a tremendous amount of effort into creating a space in which creative magic can happen.
Lesson 9: Think like a Boy Scout
Be Prepared — Boy Scout Motto
At any conference or event such as Startup Weekend there will always be a need for extra outlets, connectors, etc. Simply having a small power strip is a great way to make friends and meet new people. Bring one; you’ll be glad you did.
On the more expensive end of things, a micro-projector or access to a large monitor is useful to have people looking at the same thing without needing to huddle around a laptop. Also, it’s good for practicing presentations.
Lesson 10: Be Excellent to Each Other
A lot of what I’ve learned this last weekend has to do with people and small group dynamics. I was fortunate to be a part of a great group of people.
I discovered, too, that over the years I’ve picked up a lot more than I realized about groups, process, and business. I don’t know much about marketing or making a business plan, but I realize that I have learned a lot about people and how they work. I find that I have a better handle on project management than I might have thought, too.
In the course of Startup Weekend, there will likely be some conflict. A certain level of conflict can be good — if nothing else a better product can emerge through synthesis and refinement. However, it’s still imperative to be committed to each other and to the team.
Many cultures have a version of “Do unto others as you would have them do unto you” — in some ways I think that one of the most profound ways of expressing it is espoused by the most excellent Bill & Ted. On the surface they might appear foolish, but:
The Fool is the spirit in search of experience. He represents the mystical cleverness bereft of reason within us, the childlike ability to tune into the inner workings of the world. The sun shining behind him represents the divine nature of the Fool’s wisdom and exuberance, holy madness or ‘crazy wisdom’. On his back are all the possessions he might need. In his hand there is a flower, showing his appreciation of beauty. He is frequently accompanied by a dog, sometimes seen as his animal desires, sometimes as the call of the “real world”, nipping at his heels and distracting him. He is seemingly oblivious that he is walking toward a precipice, apparently about to step off. — The Fool
Lesson 11: Be Pragmatic
The perfect is the enemy of the good. — Voltaire
It is important during Startup Weekend to dream high. However, there are only 54 hours. There is only so much that a group of humans can achieve in that time. So it’s important to both strive for the best yet accept limitations.
Lesson 12: Have fun
It’s a lot of work. A lot of stress. But in order to be successful, it needs also be a lot of fun. If it’s not fun, you’re doing it wrong.
Thank you to everyone who made Startup Weekend possible. I’d like to thank the sponsors, organizers, mentors, judges, and other teams. But most of all I’d like to thank my teammates. You are all excellent!
Oct 29
First do no harm
If you can, help others; if you cannot do that, at least do not harm them. — Dalai Lama
Over the past couple of years I’ve been trying to put into words my thoughts regarding the type of company where I’d like to work. There are companies I would not consider working for — their ethics and/or business model are radically different from what I consider good or right.
“Evil begins when you begin to treat people as things.” — Terry Pratchett, I Shall Wear Midnight (Discworld, #38)
- I don’t want to work for a company which takes advantage of others.
- I don’t want to work for predatory companies or ones which don’t treat their employees well.
- I don’t want to go home at the end of the day feeling slimy or otherwise compromised.
- I don’t want to work for a company which treats people as things.
I think technology really increased human ability. But technology cannot produce compassion. — Dalai Lama
I’ve had the fortune to work at a not-for-profit for the last nine years. Not being profit-driven means we’re not out to gouge people. Not that there’s anything wrong with seeking a profit, but seeking profit for the sake of profit leads to evil.
At the same time, without the relentless drive for profits it’s easier to focus on customers and service.
- I don’t want to be associated with a company that views profits more valuable than people.
Don’t be evil. We believe strongly that in the long term, we will be better served — as shareholders and in all other ways — by a company that does good things for the world even if we forgo some short term gains. This is an important aspect of our culture and is broadly shared within the company. — Larry Page, Sergey Brin, Google IPO
A lot of people have scoffed at Google’s ideals in recent years. Nevertheless, it’s an admirable goal. I wish that more people or companies chose not to be evil in their day to day interactions with others.
The Agile Manifesto has brought a lot of change to Software Development. Other fields have recognized its inherent value and rewritten it to fit their circumstances.
I’d like to present a first draft of a manifesto for business.
A Business Manifesto
We are uncovering better ways of running a business and helping others do it.
Through this work we have come to value:
- People and interactions over profits and prestige
- Quality service over quantity of service
- Customer relationships over contract negotiation
- Flexibility over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
In a nutshell I want to work for a company which values people — both inside and out of the company. I want to work where people strive to do things right.
Trust yourself. Create the kind of self that you will be happy to live with all your life. Make the most of yourself by fanning the tiny, inner sparks of possibility into flames of achievement. — Golda Meire
At the end of the day, I want to work for a company which isn’t evil. When I go home, I want to be able to look in the face of my daughter and not have to make excuses for the work that I do and the effect it has on others. And I want to be able to look myself in the mirror, too.
Be excellent to each other and Party On Dudes — Abraham Lincoln in Bill and Ted’s Excellent Adventure
