Managing Too Much Complexity

"Any fool can make things bigger, more complex, and more violent. It takes a touch of genius - and a lot of courage - to move in the opposite direction" - Albert Einstein.

Why is managing an enterprise portfolio so hard? It is hard because we make things too damn complicated. We have fragmented our organizations to the point that getting anything done is nearly impossible. We are so focused on optimizing the subcomponents of our organizations we forget about making a cohesive whole.

Let me give you an example. It is pretty common for development organizations to create silos of people organized around a particular skill set. You have a manager for the BA team, the QA team, one for the Java developers, and another for the Cobol folks. The idea is that we want the best group of QA people we can possibly get. We want to establish best practice and consistency. We want managers that know how to train and develop QA talent.

Now what happens when we need to get a project done? Well... we reach out to each of the functional managers and request resources. Sounds reasonable until you realize that each of these resources is not required on the project 100% of the time. I only need that QA person part time during a few specific months of the project. What does that person do with the rest of their time? They work on something else.

You have just established your first two dimensions of complexity.

When you look at a project portfolio this way, you create a series of interdependencies between projects that are very complicated to manage. The likelihood of any one project, deliverable, or task landing exactly when it is planned is generally pretty low. Any deviation from plan creates a chain reaction through the portfolio that decreases the likelihood that anything will get done when you planned.

Most companies are not content to stop with two dimensions of complexity, they want more. I routinely see organizations trying to manage 5 or more. Let's explore some of these dimensions before we talk about how to detangle this mess.

Organizational Complexity

This is the functional silo arrangement that we already discussed. This results from a desire to optimize for individual skills over team work or product alignment.

Project Complexity

Project complexity results when we create dependencies between projects due to tight coupling of resources or shared components. Projects are essentially investment decisions. Tight project coupling limits the organizations ability to take advantage of new market opportunities.

Team Complexity

This one is related to Project Complexity in the sense that it results from building teams of matrixed resources. Often organizations will see value in building teams but fail to align them with a particular initiative or product line.

Architectural Complexity

To some degree, most companies take advantage of reusable system components and other shared services. I am loosely referring to this set of shared components as the Architecture of the system. Unless you have total shared ownership of the code, it creates another dimension that has to be managed.

Product Complexity

Product complexity results from matrixing the product roadmap across any of the other dimensions of complexity. This type of complexity results from lack of consistent investment in the product line.

These are the five I see most frequently but please let me know if there are others I am missing.

Okay… back to our original question. Why is managing an enterprise portfolio so difficult? It is because we are trying to manage across too many dimensions of complexity. We are creating too many tightly coupled entities that all have to operate in perfect synchronicity.

We know this is complex and nearly impossible to manage so what do we do? We create huge bureaucracies to deal with the complexity. We have to make sure that people are in the right place at the right time working on the right things that all the interdependencies are managed and the schedules are kept up to date. Phew!

This is a crushing amount of work. Even if you can model it, the people in your organization either can't or won't invest the time to understand it. It is a complex abstraction of reality but it isn't real. People don't believe it, don't have any ownership of it, and therefore it gets ignored.

People will pay lips service to the model until it gets to be crunch time.  When crunch time hits, people will simplify your organization for you. They work on their number one initiative and everything else will get put on the back burner. If your lucky, your organization will deliver its highest priority, but everything else will suffer.

The Solution

Let's get intentional about how we manage complexity in our organizations. I believe that a well run organization can deal with maybe two dimensions of complexity. Most management systems can adequately deal with two dimensions of complexity. It is something people can get their heads around and own.

So… what does this mean? It means that you have to pick two dimensions and align the rest of the variables in accordance with the two you choose to manage.

For example, you could choose to align your Organization, Team, and Architecture on one axis and align your Product and Project structure on the other. An alternative might be to align your Organization, Team, Architecture, and Product while keeping Projects on a separate axis. Most agile literature assumes one axis with all the dimensions of complexity in alignment.

This is part of what makes Agile so hard to implement in many companies.  Its just not how we are currently structured.

Every organization is different and operates under different constraints. You will have to decide for yourself what dimensions of complexity you business can really handle. The goal is to reduce complexity than to try to manage it.

Good luck!

Comments

[Jurgen Appelo]

Hi Mike,

Actually, you are talking of complicatedness, not of complexity. You see, people usually cannot reduce complexity of systems, but they can reduce complicatedness. For example:

The wiring on an aircraft is complicated. To figure out where everything goes would take a long time. But if you studied it for long enough, you could know with (near) certainty what each electrical circuit does and how to control it. The system is ultimately knowable.

So complicated = not simple, but ultimately knowable.

Now, put a crew and passengers in that aircraft and try to figure out what will happen on the flight. Suddenly we go from complicated to complex. You could study the lives of all these people for years, but you could never know all there is to know about how they will interact. You could make some guesses, but you can never know for sure. And the effort to study all the elements in more and more detail will never give you that certainty.

So complex = not simple and never fully knowable. Just too many variables interact.

Therefore: airplane = complicated, air flight = complex
Therefore: code = complicated, software = complex

I blogged about it here:

http://www.noop.nl/2008/03/complexity-vers.html