Start of line of thought

Last week I watched “Managing under Conditions of Uncertainty” by Dave Snowden from Agile Latvia. Starting at 1 hours 7 minutes in he says:

…we live in a dominantly complex adaptive system…in which we can influence direction of travel but it’s very difficult to re-engineer…and that requires us to take a service relationship in which we do smaller things faster and more interactively…we check them regularly to see if they’re still stable, we destabilise them as necessary…we allow a complex set of developments to continuously evolve to meet the need of service users…we stop creating a manufacturing product because software is not a product, it’s a service.

(emphasis mine)

This resonated with my belief that software development should be continuous. I then saw this slide from Richard I. Cook’s talk at Monitorama, by way of Jez Humble’s tweet, which says:

More contributors releasing software asynchronously into a production system.

Which again resonated with my thoughts on continuity and then later that same week I had an interesting conversation about technical governance from which my take-away was:

How can we, non-intrusively, ensure that teams not only proactively think about and identify their current areas of investment, technical debt and the overall state of their software but that they also continuously refresh their knowledge about it in a simple and shareable way to facilitate open discussion about choices and trade-offs to at best avoid and at worst clearly see when our latest shiny tool has become a non-adaptive legacy behemoth?

I then realised I could claim to have, in a way, adressed this.

An evolving system overview

I recently spent almost four years in a team building software, as a service, that was made up of a lot of interconnected, independent, components. While we documented each component independently we also maintained a system overview which we named “Architecture Overview (Past, Present & Future)”. This was a high level diagram of the system illustrating all its components and whether they talked to each other in a synchronous or asynchronous fashion.

Example of such a diagram

A grey box is an outside system not owned or maintained by the team. All the other small boxes are components. Those that are connected a single arrow communicate synchronously those that are connected via a small “pipe” (topic) and a “stack of documents” (queue) communicate asynchronously. The big background boxes denotes domains which was also, mostly, the equivalent of a team.

One trivial but key thing about this diagram: it was maintained as a Google Drawing which the whole team edited and everyone in the company could view. By turning on automatic publishing of the diagram we could include an auto-updating image version in Wikis and Markdown documents. Linking those images back to the Google Drawing made for simple editing if one spotted something not accurate.

Visualising evolution

Now to the interesting point of this diagram: its colours. Green boxes indicate a component that is currently, remember that this is a live diagram, considered stable. Orange ones denote components currently being either developed or refactored. Purple ones are conceptual components that we, at the time, were discussing or contemplating about introducing. Red ones denote deprecated components in the process of being phased out and once their runtime was deprovisioned they where removed from the diagram and their repository archived.

In hindsight my view is that the structure of the system as captured in the diagram allowed the individual developer to easily zoom in and work on an isolated component, “walled off” by unit tests, while it also allowed us, with little effort (<60 minutes every third week), to facilitate a discussion that caused everyone to think more holistically about the system as made up of components while, at the same time, maintaining an up-to date picture not only of the system as a whole but also of its current evolutionary state. The creative chaos of having some 15 individuals in the same diagram at the same time is also very funny, especially when the ones not contributing at the moment spend time pasting animated gifs into the diagram.

Which brings me to my current line of thought.

Generations, revisions and versions

If software is a service rather than a product and is continuously changed by individuals making independent asynchronous changes to parts of the service: what then is a version?

The system as a whole can’t make a complete and consistent shift in one atomic operation.

My previous team deployed components continuously and independently from each other. We did several major transitions both in terms of the underlying infrastructure as well as cross cutting concerns such as authentication without downtime. Therefore I view the system as, at any given time, being made up of a mix of generations, mostly a majority of one generation, usually the then current one. While transitioning from the current generation to the next generation the proportions gradually change until the system, for a period of time, is completely made up of the then current generation, the previous generation having been phased out and the next, future, generation not having been started. But more likely is that the system is in a constant transition between generations.

At any given time, regardless of whether the system is made up of a mix of generations or solely of one generation, the system consists of a number of components each at their own revision and most of them exposing some form of API, whether synchronous or asynchronous, of a given version.

Current idea

If a system is composed of multiple, somewhat, independent components it has no version but consists at any given time of one or more generations. Any given component is, at any given time, made up of one revision (or in flux between revisions in the case of canaries) and the components themselves expose a number of APIs that each are of a given version. An API can’t change its version without the revision of the component exposing the API also changing (albeit the actual switch might be a outside toggle rather than a deploy). A revision of a component can change while still exposing the same API version and belonging to the same generation of the system. The system can’t fully transition to a new generation without each component being deployed at least once, thus changing each components revision.

So, such a system is a service composed of overlapping generations made up of revisions exposing versions.

Remapping this to the diagram we used in my previous team I’m wondering if perhaps the colours for components should instead denote generation and perhaps be limited to, for example, three (previous, current and next) depending on what level of complexity the team and system can handle.

Color coding generations

In transition between previous, current and next

Color coding generations example 1

Transitioned to current and next

Color coding generations example 1

Cycled and transitioning to previous, current and next

Color coding generations example 1 (Note the changed meaning of colours in the legend)