Development Is Complex, Not Complicated

Published 21 Nov 2020

Understanding the difference in meaning between complex and complicated puts you in the right mindset to succeed in your development practice. The difference is significant, and failure to understand it is at the heart of many unsuccessful projects. I have found that staying "complexity conscious" has helped guide my decision making and avoids an over reliance on rules and bureaucracy.

The stimulus for this line of thinking was Brave New Work by Aaron Dignan. This book looks at how current models for organisations, based on the factories of 100 years ago, prevent people from doing their best work. Dignan argues that organizations aren't machines to be predicted and controlled but complex human systems full of potential waiting to be released through autonomy, trust, and transparency. The book is an inspiring read and chimed really strongly with my own frustrations about working life and wanting to find a better way. Getting clear about the nature of complexity is one of the key pillars in the book:

Contrary to popular opinion, among people who study systems theory, "complicated" and "complex" are distinct words with precise meanings. The engine of a car is complicated. A complicated system is a causal system - meaning it is subject to cause and effect. Although it may have many parts, they will interact with one another in highly predictable ways. Problems with complicated systems have solutions.

...

Traffic, on the other hand is complex. A complex system is not causal, it's dispositional. We can make informed guesses about what it is likely to do (its disposition), but we can't be sure. We can make predictions about the weather, but we cannot control it. Unlike complicated problems, complex problems cannot be solved, only managed. They cannot be controlled, only nudged. This is the domain of the butterfly effect, where a small change can lead to something big, and a big change might barely make a dent.

...

Complex systems are typically made up of a large number of interacting components - people, ants, brain cells, startups - that together exhibit adaptive or emergent behaviour without requiring a leader or central control. As a result, complex systems are more about the relationships and interactions among their components than about the components themselves. And these interactions give rise to unpredictable behaviour. If a system surprises you, or has he potential to surprise you, it is likely complex. Software is complicated. Creating a software startup is complex.

Appreciating and dealing with the lack of determinism in complex systems is what Dignan calls being "complexity conscious". Whilst software is complicated, the people and problem domains that it is built by and for are complex. Not being complexity conscious is why so many development projects go wrong.

What Happens When You Are Not Complexity Conscious?

Most organisations are obsessed with creating and enforcing processes. They are attracted to the idea that there is one right way to do a thing. When a project goes wrong it must mean that the process needs to be amended, to have more detail added, or a new process created. This creates what Jason Fried called "organisational scar tissue", ever more red tape that slows every project down because something went wrong once. This relies on the assumption that development is complicated, you just need to find the right set of instructions and you can safely navigate through any project.

But of course every project is different. Each individual brings their own unique background. The problem domain has its own unique challenges. Technology constantly evolves. All of those elements interact in a non-deterministic way. You can't reduce it to a paint by numbers instruction set that any team member can use to deliver. You can't endlessly swap people in and out of a project (usually using that pernicious and dehumanising word "resources"). We are not factory workers turning out widgets. Trying to impose a monolithic process just leads to a sense of complacency from management, the process will take care of it, and when it doesn't they will usually blame the team for not following the process. The reality is that you cannot process your way through complexity.

Complexity Consciousness And True Agility

So if processes aren't the way to deal with complexity, what is? Well, I think the key insight is that a complex system needs to be constantly monitored. You cannot fire and forget. Of course you need to try and make long range plans, but those need to be constantly reassessed and you need to accept that there is a predictability gradient. You can be confident about today, maybe tomorrow, possibly the end of the week, but after that don't count on anything for sure.

Monitoring metrics need to be rich enough to capture as many of the different factors at play as possible.

Is something starting to sound familiar here? It should do. This starts to sound a lot like agile software development, epics and sprints. Indeed, Dignan talks about agility (with a small "a") as a key way to cope in an uncertain world. However, I think a lot of organisations that run agile processes are just replacing one dogma with another. They still treat the process as the rigid framework, which misses the truth at the heart of the agile manifesto. As a reminder the manifesto states:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

I don't think you could come up with a better set of principles for dealing with complexity, but I think most organisation miss the point and treat agile as a fixed process. The manifesto does not say you must work in sprints. It does not say you must have daily standups. It does not mandate a scrum master. These might all be good starting points for your team practice, but they get treated as gospel, if you don't do them then you aren't doing "proper" agile. This is daft. They key to improved outcomes is to constantly assess what you are doing and change your practice accordingly. You need to be able to respond to change, to adapt the way you work on this project right now.

Agile And Continuous Participatory Change

The value in working in sprints, and holding retrospectives, is that it creates a space to do this reflection, to assess how things are going and adapt the plan for the next sprint. However, most agile teams focus on the software in these sessions and not the process. This is missing opportunities  to improve the project. Why not spend part of these sessions on the process itself? Is the sprint duration right? Should you look at a different way of assigning tasks? Should you carve up responsibilities differently?

A key aspect of this is giving the team the autonomy to consider and adapt the way that they work. This is what Dignan calls "continuous participatory change". Dealing with complexity means moving beyond bureaucracy. The many moving parts of a project make it difficult for a manager to step in and solve the tensions. Typically this just ends up in seagulling. The individuals engaged in the process are the best ones to understand the tensions. Giving them the confidence and ability to suggest modifications to practices is the best thing you can do as a manager. As Dignan writes in a blog article on organisational debt:

You can offer your team(s) a regular mechanism for editing the organization itself. If individuals have the power to recommend changes to their own roles and rules, and teams have a process for vetting and shaping these proposals, something extraordinary happens: the firm gets smarter every day

A cynic would say that this will just lead to developers "going rogue" and you'll end up in a bigger mess, but I'm not advocating no process, no team norms. I'm just arguing that your ways of working needs to be constantly assessed. You need to be mindful of tensions and be prepared, as a team, to change the way you do things to reduce those tensions. You should have standards and processes, but they should be seen as defaults that can be modified. As Dignan says, this allows the team and the organisation to get smarter.

Complexity As A Guiding Metaphor

So understanding complexity helps to motivate a more participatory and evolving organisation. Sometimes a metaphor or a core idea can act as a guiding light, a way to steer your decision making, and that's the real value I see in keeping complexity in mind. 

I find these images, and the striking differences in their causality and predictability, to be incredibly useful in deciding on the right action. It helps avoid the temptation to try and process your way out of a problem, and forces you to address the need for vigilance and focus on your ability to react and change.