Thursday 25 September 2014

Microservices Hackathon September 2014

Practise improvisation daily. Practise making these harmonies or chord changes connect in a flowing, melodic way when improvising. - Horace Silver, The Art of Small Combo Jazz
We held our second microservices hackathon in London on 13 September (Programmers' Day, as it happens). Wow!

All these folks (and more who didn't make the picture - about 15 in total) are working in teams on a single application - a wacky cross between Tetris and Scrabble. We had teams coding in Java, Python, and Javascript, all communicating in the combo style. The result was a game server capable of running at least five simultaneous games without breaking a sweat, with a high-score board and very responsive gameplay - see Dan's video to get a sense for the game. OK, you only scored points for one word ("HORSE") but it was still a lot of fun to play. To see the code, visit the hackathon repo.

As with our previous hackathon, we learnt all kinds of things about how to run a good hackathon and about the microservices style - for example:
  • RabbitMQ connections, channels, exchanges, and topics can be confusing! We use only a simple subset of RabbitMQ's features - just pubsub really - and some of these elements are redundant for our purposes, hence troubling to newcomers. The combo API helped some folks, and we'll try to give an initial intro and provide examples to ease the way next time.
  • Speaking of examples, we had several ideas on how to help participants master the tools faster:
    • We should provide a pointer to previous hackathon repos to help people get started faster.
    • We should build a sample app to help teams see how their components could look.
    • We should provide a sandbox where teams can publish test facts. In several cases, a test publication by one team crashed another team's under-development component - ten points for robustness testing but minus a lot for inconvenience and surprise!
    • After the event, Tom Marks contributed a Python example client.
  • One person (Stans I think?) set up a Slack chatroom which was very helpful for exchanging fact formats and announcing events. We'll be sure to have this up before starting next time.
  • The process of setting up and co-ordinating the teams was lots of fun as always. We came up with some good ideas for collaboratively designing the facts and behaviours:
    • Tom put some of the facts on the whiteboard, together with the teams that were interested in them - see the bottom half of the above picture.
    • sleepyfox had a nifty idea we'll try out next time - creating CRC cards for each component and having the teams pass around notes representing the facts to see where the collaboration works (and where it doesn't!) 
  • Finally, we can't forget to have nametags next time, for forgetful folks like me.
Our next hackathon is coming at the end of October - feel free to join us!


Monday 18 August 2014

Microservices Hackathon July 2014

When playing in a group, you must unify and harmonize with the group and with all of their concentrated efforts in order to project that group in a most favorable light. - Horace Silver, The Art of Small Combo Jazz
Fifteen developers, eight different languages, and plenty of pizza - that was the recipe for the first Microservices Hackathon in London last month. I had organised it but wasn’t sure what the result would be - in fact we managed to learn a whole bunch of new technologies, co-operate successfully to build a variety of components in several languages, and construct a simple MUD using combo-style microservices. To learn how, read on!

After a couple of experiments writing microservices systems on my own, and a well-attended talk at SkillsMatter, I’d decided to try out combo-style microservices with a larger group at a Saturday hackathon. My goal was to get everyone writing code, learning new things, and trying out real polyglot programming using the independence afforded by the microservices style.

I started by setting up the Meetup group and was surprised and pleased at the vigorous response (with no marketing at all). Once I knew people would really come spend a Saturday with me hacking microservices, I got going on creating the code we’d use to communicate. In accordance with my ideas about the combo style, this consisted of a RabbitMQ instance for pubsub and a PostgreSQL installation for the fact database, together with an archivist component to record published facts in the database. I also built a sample application and included a deployment mechanism (which we turned out not to need). I installed all of these on Amazon Web Services and opened all the firewalls to the world (fun to ignore security for once).

As everyone arrived, I noted that we had developers whose native languages were PHP, Clojure, Go, Python, C++, C#, Java, and Javascript. The final product featured just four of these languages - the others proved too difficult to get running with PostgreSQL and RabbitMQ, with Rabbit being particularly challenging (the poor Clojure chap spent almost the whole day trying to get his RabbitMQ library to work). Even for those four languages, most folks took 1-2 hours just to get connected properly to the Amazon instances.

Once we had all managed to publish, subscribe, and read data in at least one language, we went on to plan our project for the day. After a brief debate, we settled on a multiple-user dungeon as our target, and split our work into three components: one in Java to create and manage rooms, another in C# and node.js to handle movement among rooms, and a third, also in Java, to provide a user interface. Later we added a fourth component in Python to report on the locations of all users, and an extension in Javascript to show an interactive map of the dungeon. A MUD seemed like a particularly good fit for the subject of the day’s hacking; it was simple enough to grasp quickly, decomposed nicely into independent, business-meaningful components, and lent itself naturally to the definition of a ubiquitous language.

Everyone settled in to a quiet afternoon of coding. Other than helping some of the teams to make decisions about what facts to publish and when, I didn’t have much to do besides order the pizza and eat some of it. I was pleased to see that we appeared to have achieved component independence, one of the main goals of a microservices architecture - that is to say, once the co-ordination language was agreed, the individual component teams were free to get on and implement without worrying about orchestration or API design.

Toward the end of the day, I was pleasantly surprised to see that all the teams were close to finished. Because all of them had already been publishing and subscribing throughout the day, integration just consisted of cleaning up the fact database and using each component in turn - entering some rooms, doors, and descriptions using the room manager, issuing move commands using the command-line UI, and observing the results - either successful movement to a new room, or failed movement when no door was present, thanks to the movement component. Everything worked smoothly including a nice dynamic update to the room map. It was really impressive to see that we’d managed to create so much in just a few hours, even though most of us didn’t know the others’ languages!

Some interesting observations along the way:

  • I had expected everyone to want to deploy code onto servers, and was ready to spin up more including Windows instances if needed. As it turned out, with good wifi and fast laptops, it was easiest just to have each team run its own code locally - in essence we had a five-server cluster (the one I set up at AWS running the archivist and RabbitMQ, plus one machine per team in the room). This means we didn’t get to practise using any of the clever tools for microservices deployments, but the gain in efficiency was definitely worth it.
  • The big monitor in the meeting space was really nice - we particularly found it helpful to run a real-time view of the fact database, so everyone could see what was being published and picked up (or not picked up) by their components.
  • Clearly the most painful bit was connecting to RabbitMQ and PostgreSQL. I was disappointed as I thought these were mature technologies with high-quality libraries in most languages - perhaps we just didn’t know enough to pick the right ones. Next time I plan to provide an HTTP shim for both with verbs like “publish” and “query”, so no one has to struggle for very long with a recalcitrant library. (Well, COBOL folks might have to hunt around for a good HTTP library, but I’ll take that risk.)
  • The hackathon format didn’t lend itself to stress-testing the design. Not only did we not have time to create a well-documented, thoroughly tested product, we didn’t get to some of the communication and evolution challenges that microservices critics have noted - we didn’t see any emergent behaviour, didn’t try to evolve our language or even our components very much, and we didn’t worry at all about latency. I think this is OK since we weren’t really trying to do such a stress test - we just wanted to learn about this new way to write polyglot, robust code.

Nearly everyone said the day was well spent and expressed interest in attending another hackathon, so I’m planning to hold one in early September. Join the Meetup group to find out when that is!

(Thanks to my employer Osper for sponsoring the event. If you’re interested, you can read about some more cool stuff we’re trying at Osper, and of course we’re hiring.)

What Are Combo-Style Microservices?

Jazz is not a background music. You must concentrate upon it to get the most out of it. - Horace Silver, The Art of Small Combo Jazz
In this first real post, I'd like to introduce the particular style of microservices I want to write about here, for which I'm trying out the name "combo-style microservices". I reserve the right to think of a better name in the future!

To understand this style, it's helpful first to look at the emerging standard definition of microservices as defined in Fowler and Lewis's seminal post. They describe building an application out of many small components, each independently developed and deployed, making service requests from each other over a simple routing system (HTTP or lightweight messaging). Components are naturally called "services" since each provides a menu of services to other components, available over API calls; by one means or another, e.g. through service contracts, these API calls are published and maintained so they can be used reliably. Each service provides a full business capability including a user interface and data storage, if those are needed.

I liken this standard definition to a large orchestra all playing a symphony together - it requires a high degree of co-operation, communication, and orchestration. The co-ordination may be centralised in the person of a conductor or distributed as in a self-conducted orchestra, but in any case, the musicians must come to clear agreements about many things: who is in which section, how fast to play, whether to include all the repeats, and much more. And while performers can make their own decisions about matters that affect just them - say, when to turn the page in the score - no member can make even the slightest externally audible change unilaterally. If the oboist decides to play fortissimo instead of piano in bar 12 without telling the other players, the whole performance is likely to be ruined as notes are drowned out and players miss their cues. Similarly, while a Fowler/Lewis microservices system can allow its services to make whatever internal choices they like (programming language, libraries, data storage), it has to enforce strict control and testing on externally visible changes, such as changing the return type of a service call or removing it altogether.

Consider instead a jazz combo, a group of musicians who form a much looser combination with minimal co-ordination. They agree on a few basics - the key and the chord progression, for example - and then improvise rhythm and melody separately based on these. Minor audible changes, such as an embellishment of a theme or a new countermelody, are expected and welcomed by other players, who ignore them or respond to them based on their own view of the evolving music. Even significant modifications like bringing in an additional musician to "sit in" are acceptable and add to the overall musical experience. Because its members have so much freedom to improvise, no two performances by such a combo are the same.

In a combo-style microservices architecture, we strive to replicate the jazz combo's flexibility and independence. We retain the notion of building our application from many small components, each developed and deployed independently and providing a business capability - but the components no longer make API calls to each other. Instead, each component publishes one or more business-meaningful facts (such as "Billy has purchased Legos for £20") using a ubiquitous language, and other components subscribe to those results they are interested in. The collection of all results is stored in a fact database that can later be searched ("find all toy purchases in the last five minutes for over £20”). If you like, think of the components as micro-applications, each one performing a useful small task like "compute sales tax" or "authenticate user", and the whole producing the required behaviour for users. The musical combo co-ordinates its work through the basic agreements on key and chords, which change little if at all during a single musical piece, just as our microservices combo co-ordinates through the common and (we hope) relatively stable language of the domain; in both cases, individual participants can vary their behaviour quite substantially without disrupting the activities of others.

There are many obvious objections to the practicality of such a system: it will necessarily have some emergent behaviour, the language or the domain may not be sufficiently stable, the publish/subscribe model may not perform. I shall try to address some of these in future posts (and perhaps will convince myself that the ideas need modification!) but I hope this explanation has helped make clear where I currently stand.

(Thanks to Antony Marcano for inspiring the jazz-and-orchestra comparison.)

Edit: I noticed later that the Fowler/Lewis microservices post has one reference to an earlier Fowler article on Event Collaboration, which describes an event-based design similar to the combo style. (An interesting difference is the lack of a fact database - an exercise for the reader is to determine how the example at the end of the Fowler article is simplified if you introduce one.) This is encouraging - maybe I'm not alone in thinking combo style! - but on reflection I still think it's fair to say that the microservices post largely assumes that components will call each other via APIs, with references to consumer-driven contracts, published interfaces, and coarse-grained API design.

Tuesday 17 June 2014

Microservices talk!

Gave a talk at SkillsMatter on Microservices. Here are the slides, and here is the video .