The DNA of CHAOS. Why software projects, political negotiations and football games all come up short on occasion.

Why Business analysts, Product Owners, negotiators, political reformers, scientists and many others are doomed to repeat the same mistakes until they drown in in their failures.

  1. Most people in a work situation, and indeed in most situations, live in a consciousness that is deliberately falsified to reflect (a) what they believe is expected of them by peers, and (b) what the asker of a given question wants to hear. This is a conditioned response and nobody’s fault.  Terms like “Group Think” attempt to explain it, but none really explain the phenomenon well and the long version is just beyond our scope in this piece. Market researchers are trained to try and avoid it, Politicians and advertisers learn to harness it. If you have the role of finding out what  a business wants to do so you can get the software designed to support, it, you will be faced with the same  aggravated problem. Asking and listening simply won’t get the job done.
  2. For reasons I don’t claim to understand fully, most people have a favourite solution in their minds for most problems they are aware of. Expensive  housing – they may say; “Stop immigrants coming in”.  The call centres is too expensive and eating our profits- they may say; “ Close the call centre and let them use Twitter”. Always there is an obvious political, or financial gain for the responder from this solution. If it is a senior executive, or politician, there will be cleverly argued points and tables of figures that make her pet solution look like a no-brainer.
  3. Michael D Cohen  coined the phrase “Garbage Can Theory” to describe this phenomenon which permeates politics, business, the playground and just about every area of human life crippling every endeavour to improve our lot century after century. If you are trying to get your proposal through programme and investment boards, or trying to agree a solution closer the coal-face you will drown unless you understand this phenomenon and take steps to work with it.
  4. Many researchers and writers have agreed strongly that the human brain makes virtually all decisions in a matter of time between milliseconds and 1 to 3 seconds, but then spend enormous amounts of time and effort researching in order to prove their decision was correct. Most people are entirely unaware that they do so.  This of course is not news to Scientific researchers who often publish the theory before beginning on the research.  In the academic world where we have to start somewhere this can be a useful approach. Product managers and entrepreneurs also start with the theory and are difficult to budge even when wrong, yet these leaders are necessary to get something started at all.
  5. People are conditioned to follow a strong leader and will convince themselves of their support for any theory put forward by this leader. This is largely an extension of  point one, but worth being aware of all the same when venturing into any group to arrive at a shared view.

These Five forces combined thwart just about every effort to form a good business strategy with technology in mind, agree a solution and still be in agreement when the solution is completed to specification even a few months later, let alone after a year or more. Waterfall and other methods of managing the software delivery process all rely on a simplistic view that if you gather enough requirements from users and stakeholders, you can then haggle over priority and build the killer solution to solve everyone’s problems. In reality, there is rarely any agreement, there is even less concern over business outcomes and near- total focus on the many garbage can solutions. Nothing this project could deliver is ever likely to meet the satisfaction of more than a small proportion of stakeholders, the others are always going to be loud in their dissatisfaction and only by sheer luck is their ever going to be serious business benefit.  This project, however well the techies perform will be added to the list of “failures” and the cause will be placed at the CIO’s door.
Failure to focus on goals Developers and technical architects are driven by developing exciting new toys. That is their package in the garbage can. Techies ultimately want and are entitled to, clear definitions of what they must build.  They then build and test precisely what you asked for and give it to you working  very well. Few people would try to argue with that, let alone succeed.

The Midas Problem

The problem is that what you asked for does not always address the problem you needed solved and in fairness the developer or indeed the CIO is not to blame. We call this the Midas problem because it is explained well by the man who wished everything he touched would turn to gold. We all know how that worked out.
There is one technique that all accomplished negotiators agree on whether brokering peace in a war zone or negotiating a business contract. YOU MUST KEEP THEM FOCUSED ON GOALS NOT SOLUTIONS! If you don’t, the garbage can will take over.
One cause  of the problem is not using the effective solutions we already have,
The need to focus on goals rather than solutions is not  new to the software industry by the way.  MSP, Prince2 and TOGAF are frameworks I am personally very familiar with, all of which, when used correctly, focus on outcomes required, the solutions that will best deliver those outcomes and the details of each solution, in that order. BABOK and the ISEB framework for business analysis also follow these same principals and are very clear about it. The problem is that few people make any attempt to use their training effectively, the training is rushed and focused only on answering multiple choice questions to gain a certificate and of course the majority of people involved in the software business have no training at all.

This same scenario, by the way, applies equally to Scrum. Scrum solves some of the problem Scrum attempts to alleviate the problem in a different way and with a similar level of success. Scrum is invented and owned by the technical community as a direct response to these issues plus the difficulty of communicating the complexity of technology with business people. The Scrum team wants only to be given an unambiguous requirement so they can do what they studied for, build great software. They place the onus on the business to appoint, or work with a Product Owner who will be seconded to their team and take all responsibility for the definition of the Product/System.   Communication issues are solved by very short Sprints of just two weeks typically and using prototypes and working software to describe things to the product owner in place of documents. Initially this arrangement solves the problems for the Scrum team, there is no ambiguity about what they must build, they do as much work as the Product Owner wants and stop when he is satisfied. There is no room for dissatisfaction with the IT team. Likewise for the Product Owner, if he is unsure, he can build something , test it on real world scenarios and then make adjustments until it delivers the right outcomes or scrap it altogether.
Inspect and adapt solves problems with lack of knowledge, but also masks others Scrum tries to tackle the strategic questions by replacing strategic decisions and documents with Inspect and adapt. For online products and for certain internal systems, this works extremely well because you can build the minimum, get it working and in use, learn from success or failure and add a little more, remove some, or start again. The risks are lower, failures are smaller and less expensive and the problems with groupthink, daydreaming and garbage cans are alleviated by an early sobering splash of cold reality that comes with release of a Minimal Viable Product. What is often wrong with this approach and fatally dangerous, is the lack of a strategic layer to the thinking process. Inspect and adapt is a close cousin of the OODA loop as defined by Colonel John Boyd. The principal is very simple that in a dynamic environment, he who has the most clear view of the situation always wins given a reasonable ability to make decisions.  Pilots flying inferior jets were seen to consistently win dogfights  because they had a better view from the cockpit and could make better decisions faster.  Developing OODA  (Observe, Orientate, Decide, Act ) and teaching it to pilots made significant improvements over giving them a detailed instruction, or a faster Jet because as the military have always recognised, in the field, the operative has superior knowledge in the heat of battle and must have the right level of autonomy and confidence to decide and act. No military  has ever cut loose a person, or team, or unmanned drone to inspect and adapt and just stay out there. There has to be a clear goal, a Definition of Done, Principals that guide it in the field and a strategy for completing the mission. Then Inspect and Adapt comes into play and in the same way that the human brain itself learns,  the actions  in the field can feed all the way back to sometimes cause a change in strategy or even a revision in Principals, but  actors in the field DO NOT  GO AWOL. Principals and strategy come first and Inspect and Adapt is a response to exceptions , it is not the  plan.

The real reason why Agile/Scrum is winning friends in the world of software and gaining attention beyond these confines

I say Agile/Scrum because when most people speak of agile they are really referring  to Scrum. However Scrum has encompassed much from other Agile implementations and definitely does not hold the sole franchise on Agile methods. The first problem Scrum had to address was the great chasm between Business leaders and Technology leaders.  Various versions of the same solution see the business seconding someone to the Scrum team, or collocation of business and technical people working on the same project. This is a perennial problem sometimes referred to as silos and secondment or collocation of cross discipline teams is a useful technique that helps, but doesn’t alone solve the problems. In the case of Scrum it forces the business into making decisions for better or for worse and living or dying by them. That helps techies to get on with building stuff, but it doesn’t, of itself, help the business much with getting the strategy right. I.E they produce more stuff, but don’t necessarily solve more problems. The next problem Scrum had to address is helping the business in dealing with the unknown. The speed of change is much greater in the past few decades driven by the fast pace of disruptive technology. Organisations are faced with problems they can’t analyse easily and therefore can’t solve. They also find that when they know the problem, they don’t have an obvious solution. This is the situation Scrum was designed for in its entirety. A Scrum team can make a best guess and very quickly get a potential solution in front of customers, Inspect and Adapt and return quickly with an improved effort until the right solution is reached by trial and error and then it can be built upon until it reaches its optimum scale. The biggest problem addressed by Scrum and that one that is causing all the excitement is the almost  total  lack of leadership throughout modern businesses (the past 10 years). Few would lament the passing of the great Command and Control organisations, but with them went whatever semblance of leadership we had in organisations. In the Flat organisation leadership in name is often cited, but leadership in deed is sometimes frowned upon or even punished. In the 80s and 90s business leaders bullishly wrote books about leadership and motivation. These were charismatic people with an instinct for what made others tick.   Kenneth Blanchard, Peter  Drucker, Stephen R Covey and their like gave us the ideas and the passion to change things and build things. We somehow lost this passion and insight in the transition.  Scrum is a beacon in the dark, because it values people and their interactions over trails of evidence. It values achievement over promises and it values self-improvement  and job satisfaction. Here and there Scrum teams are springing up that capture attention by their high productivity and ingenuity and win the envy of their colleagues for their enthusiasm and Joie de vivre.  People who see a good Scrum team in action want to be part of it or learn how to make that happen in their business.  That is the real reason why the buzz of excitement continues and why Scrum experts are being invited into other areas of business.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.