Monday 18 August 2014

Microservices Hackathon July 2014

When playing in a group, you must unify and harmonize with the group and with all of their concentrated efforts in order to project that group in a most favorable light. - Horace Silver, The Art of Small Combo Jazz
Fifteen developers, eight different languages, and plenty of pizza - that was the recipe for the first Microservices Hackathon in London last month. I had organised it but wasn’t sure what the result would be - in fact we managed to learn a whole bunch of new technologies, co-operate successfully to build a variety of components in several languages, and construct a simple MUD using combo-style microservices. To learn how, read on!

After a couple of experiments writing microservices systems on my own, and a well-attended talk at SkillsMatter, I’d decided to try out combo-style microservices with a larger group at a Saturday hackathon. My goal was to get everyone writing code, learning new things, and trying out real polyglot programming using the independence afforded by the microservices style.

I started by setting up the Meetup group and was surprised and pleased at the vigorous response (with no marketing at all). Once I knew people would really come spend a Saturday with me hacking microservices, I got going on creating the code we’d use to communicate. In accordance with my ideas about the combo style, this consisted of a RabbitMQ instance for pubsub and a PostgreSQL installation for the fact database, together with an archivist component to record published facts in the database. I also built a sample application and included a deployment mechanism (which we turned out not to need). I installed all of these on Amazon Web Services and opened all the firewalls to the world (fun to ignore security for once).

As everyone arrived, I noted that we had developers whose native languages were PHP, Clojure, Go, Python, C++, C#, Java, and Javascript. The final product featured just four of these languages - the others proved too difficult to get running with PostgreSQL and RabbitMQ, with Rabbit being particularly challenging (the poor Clojure chap spent almost the whole day trying to get his RabbitMQ library to work). Even for those four languages, most folks took 1-2 hours just to get connected properly to the Amazon instances.

Once we had all managed to publish, subscribe, and read data in at least one language, we went on to plan our project for the day. After a brief debate, we settled on a multiple-user dungeon as our target, and split our work into three components: one in Java to create and manage rooms, another in C# and node.js to handle movement among rooms, and a third, also in Java, to provide a user interface. Later we added a fourth component in Python to report on the locations of all users, and an extension in Javascript to show an interactive map of the dungeon. A MUD seemed like a particularly good fit for the subject of the day’s hacking; it was simple enough to grasp quickly, decomposed nicely into independent, business-meaningful components, and lent itself naturally to the definition of a ubiquitous language.

Everyone settled in to a quiet afternoon of coding. Other than helping some of the teams to make decisions about what facts to publish and when, I didn’t have much to do besides order the pizza and eat some of it. I was pleased to see that we appeared to have achieved component independence, one of the main goals of a microservices architecture - that is to say, once the co-ordination language was agreed, the individual component teams were free to get on and implement without worrying about orchestration or API design.

Toward the end of the day, I was pleasantly surprised to see that all the teams were close to finished. Because all of them had already been publishing and subscribing throughout the day, integration just consisted of cleaning up the fact database and using each component in turn - entering some rooms, doors, and descriptions using the room manager, issuing move commands using the command-line UI, and observing the results - either successful movement to a new room, or failed movement when no door was present, thanks to the movement component. Everything worked smoothly including a nice dynamic update to the room map. It was really impressive to see that we’d managed to create so much in just a few hours, even though most of us didn’t know the others’ languages!

Some interesting observations along the way:

  • I had expected everyone to want to deploy code onto servers, and was ready to spin up more including Windows instances if needed. As it turned out, with good wifi and fast laptops, it was easiest just to have each team run its own code locally - in essence we had a five-server cluster (the one I set up at AWS running the archivist and RabbitMQ, plus one machine per team in the room). This means we didn’t get to practise using any of the clever tools for microservices deployments, but the gain in efficiency was definitely worth it.
  • The big monitor in the meeting space was really nice - we particularly found it helpful to run a real-time view of the fact database, so everyone could see what was being published and picked up (or not picked up) by their components.
  • Clearly the most painful bit was connecting to RabbitMQ and PostgreSQL. I was disappointed as I thought these were mature technologies with high-quality libraries in most languages - perhaps we just didn’t know enough to pick the right ones. Next time I plan to provide an HTTP shim for both with verbs like “publish” and “query”, so no one has to struggle for very long with a recalcitrant library. (Well, COBOL folks might have to hunt around for a good HTTP library, but I’ll take that risk.)
  • The hackathon format didn’t lend itself to stress-testing the design. Not only did we not have time to create a well-documented, thoroughly tested product, we didn’t get to some of the communication and evolution challenges that microservices critics have noted - we didn’t see any emergent behaviour, didn’t try to evolve our language or even our components very much, and we didn’t worry at all about latency. I think this is OK since we weren’t really trying to do such a stress test - we just wanted to learn about this new way to write polyglot, robust code.

Nearly everyone said the day was well spent and expressed interest in attending another hackathon, so I’m planning to hold one in early September. Join the Meetup group to find out when that is!

(Thanks to my employer Osper for sponsoring the event. If you’re interested, you can read about some more cool stuff we’re trying at Osper, and of course we’re hiring.)

No comments:

Post a Comment