This article discusses an important aspect of agile that is rarely figures in the big debate, i.e. where agile fits in the great scheme of things organiation wise. It also examines breifly  the concerns that are shared with general corporate activity.

It sheds light on  why agile came about and regularly returns to the fore only to disappear again and most of all it puts the whole cult aspect of agile under the microscope arriving at some perhaps unexpected conclusions.

Conclusion

Some people are overawed by the challenge of software development to the extent that they rely entirely  on a comforting framework to lean against, while others use their big picture perspective to manipulate the marketplace selling  their training, books and consultancy services under one disguise after another, while contributing little or nothing to the profession.

In reality there are two major challenges facing all software endeavours. The first of these is the challenge of dumning down science and motivating people to ahieve some sort of useful marriage. The scope is beyind this article but siffice to say that the internet arrived at least 25 years after it could have and is still years behind where it could be while people catch on to the possibilities. The antithesis of Moore’s Law  if you like.

The second challenge facing software projects could be summed up as defining an agreed goal and delivering on it.
in meeting that challenge the professional will follow the one very simple Software Development Lifecycle (SDLC) regardless of what methodology or framework he/she initially elected. That lifecycle can be loosely described as:

1. Identifying the organisational goal the commitment, timeframe and budget.
2.  Identifying a technical solution that meets that requirement within agreed constraints.
3. Delivering the technical solution within agreed quality constraints.
4. Delivering on the corporate goals.

The key to choosing and using methodology is to understand the challenges thrown up by a specific project and address them correctly. These challenges will be predominantly driven by corporate culture, process complexity, level of innovation, technical competence and budget constraints.

Ultimately it matters little whether you begin by creating detailed plans and evaluating them technically, or start building storyboards and prototypes. Likewise it matters not a jot whether you build it in time boxes, or spend two years coding before shipping the lot to outsourced testers, both of these approaches will have to meet all four objectives before the project is completed successfully.

There are however, certain projects that simply won’t fit a particular approach. One useful example is that you wouldn’t be advised to take an innovative prototyping and experimenting approach to air traffic control systems for example, as the consequences of error in the “trial and error “ paradigm are well known and best avoided.

The reason for the re-emergence of agile as an SDLC as opposed to a methodology is that it includes items one and four of our basic lifecycle in a single project.

Outside of Agile it is very rare for items one and four to be given very much more than lip service.  Business people are generally untrained and ill-equipped to rub shoulders with the technical aspects of this work and are understandably very frightened to make commitments and take responsibility for something they don’t understand and can’t control.  IT people are rarely equipped with the business knowledge and people skills to pull it off even if they were likely to be taken seruously by the busienss. Hence the ostrich syndrome prevails more often than not.

Agile breaks this deadlock by placing empowered people with all the skills they need and sufficient budget into a single team and telling them to make decisions and get on with it.

Agile is to the SDLC what a project is to the enterprise

“Agile places empowered people with all the skills they need and sufficient budget into a single team and tells them to make decisions and get on with it.”

Why does this statement seem radical to some people? Surely we have sufficient grasp of management in the 21^st century that we can define a task and get it done with a level of motivation and commitment within just about any corporate body.  Well it seems not.

It is well documented and understood that once an organisation grows to the point where the founder can no longer be involved in every decision, there begins a phase that either destroys it, or morphs it into a modern political and mostly dysfunctional organisation that is driven by policies, processes and politics,  with little identity and few motivated individuals willing or able to chase opportunities or to excel.
The winning strategy to survive and prosper in this corporate world is based on being out of sight and below the parapets at all times except miraculously just after a victory and of course bringing apples to the teacher. Excellence is rarely sufficient and often unnoticed.
As these organisations grow within their specialised field, they substitute predictability, buying power and economies of scale for individual efficiency and innovation and they can survive and prosper despite the negative, but predictable performance levels produced by process heavy environments lacking in initiative, or motivation. 
The problems surface when you try to use that same team with the same culture to do something different.  Not only are you asking them to step out of their comfortable process, but a successful result will often bring with it an unwelcome and frightening change of culture for them and on top of this you are asking them to make decisions and use initiative.  All this in an environment which, through no fault of their own penalises initiative and chops off heads that protrude above the parapet.

This description may sound like a condemnation, but in truth it is just an acceptance of the fact that we are human and we adapt to our environment and that to achieve things we must begin with an honest, or even pessimistic appreciation of that environment and a plan to work within it.

Enter the Project

Although there are projects used in other environments such as construction where they are in fact part of operations, for the most part projects are used as frameworks for dealing with the issues described above. In fact, even when adapted to areas like construction, they still have a key role in abstracting the task at hand from the corporate cultures of client and supplier in order to progress the work at hand in a partnership or joint venture environment.
A typical project will create a governance structure modelled loosely on a company with a senior executive a PM and board that closely resemble the familiar frameworks of CEO, Chairman and board and operate in a similar way for course of the project. This structure releases them from the corporate culture and gives them the the framework and budget to get things done independently.  Just like real companies, they work well when the team are well chosen and pulling together and less so by degrees at other times. Projects without strong corporate support are up against it unless they have a very powerful and committed champion.

Why agile then?

Traditional Projects work well for large undertakings that are sufficiently in-your- face to gain and maintain interest in the boardroom right through to conclusion. Problems get solved and genuine mistakes are not allowed to become bargaining chips.

The trouble is that most projects don’t gain that kind of attention and are peripheral to busy directors and senior executives.  Board members are sometimes not ideally selected, not terribly committed, or even there purely to be kept informed. Occasionay they are hostile.  In these circumstances many projects lose their way for want of decisions, motivation, or intervention against blockers.

This is where the agile paradigm comes into its own.  Instead of creating this large and very formal structure that won’t actually gain critical mass, the key is to choose motivated, knowledgeable and competent people at the right level in the organisation, combine them with high quality technical and support teams and a skilled agile PM while completely empowering the team to make decisions and get the job done while providing access to a senior executive as and when required.

This agile approach returns the project to a small company paradigm with motivated, committed individuals operating with a shared vision, making decisions, communicating meaningfully, pulling together and getting the job done.

The software profession at its best

Surely we have taken software engineering to a point where a reasonably experienced engineer can decide whether something is achievable, give a time-frame and budget and stick reasonably close to it. Again it would seem to be a singularly difficult challenge for the profession.

Since the late fifties, the software industry has learned and developed along a predictable path that is in some ways remarkable and yet in others frustrating.  The return of the same old questions again and again is partly seen as the rejuvenation of boring old ideas for immediate financial gain and this view is not without any truth, but it is also part and parcel of how humans increase their knowledge.

Hegel described the increase of understanding in terms of: Thesis, antithesis ad hypotheses.

In effect he suggested that learning begins with theses, some of this is then rejected (antithesis) and finally, on reflection it emerges for use and further consideration (hypothesis).
Software engineering has developed at a rapid speed through borrowing from hardware engineering, construction and manufacturing, trying these concepts, improving them and trying again. Between 1950 and 2000 the average cost per line of code in use at the US Department of Defence dropped from $10 to $ .000001 as the industry matured. The industry has progressed rapidly in engineering terms, but there’s still a long way to go and especially in terms of interface with the users and investors.

Coding time estimates

In today’s software engineering environment (as opposed to a newly formed team of programmers at random company ltd) the time taken by programmers while programming is roughly as follows:

In fact as a good rule of thumb, it is widely accepted that when a programmer tells you how long a job will take,  you need to multiply that figure by somewhere between 6 and 11 depending on his/her track record.

If this software is to be of any use it must also be documented for the purpose of maintenance and for implementation, training and user support. In a less mature team these figures would be easily multiplied by anything up to a further five.

Communication time estimates

As you build a team the time for team and project communication begins to rise exponentially as the team size grows and if change is frequent it can reach as much as 50% of all effort with alarming speed.

Propensity for change

 The cost of changing requirements/features grows rapidly according to how late in the cycle the change occurs.  This change is mostly driven by omissions and errors at the design phase and design or programming errors further along the cycle. Controlling these changes is down to excellent stakeholder communicat

Impact of change in a software project

ion and rigorous design and verification procedures.

The preceding paragraphs highlight the very topmost concerns in the simplest possible way yet it is clear even from this short commentary that providing even reasonably accurate estimations is extremely challenging and is a whole team exercise.

It is also worth noting that of the four steps defined in our underlying SDLC, the software engineering discipline only covers numbers two and three.
 Responsibility for defining the business goals and their parameters as well as responsibility for implementing the system in order to capture those benefits remains an equally challenging proposition and lies entirely with the business.
In a product paradigm, the only difference is that product management must take responsibility for customer consultation and feature design.

Software Development Lifecycles (SDLC) are over prescriptive and misunderstood.

The many incarnations of SDLC will generally expand very quickly into a complex map of steps with interdependencies. With the best intentions, they attempt to tie the practitioner into a very prescriptive set of steps, but in doing so they remove the emphasis on common sense and good communications. For the sake of simplicity, when appraising a project, I tend to simplify the lifecycle to just four goals that can be carried out in any order that serves your specific project and revisited when necessary.  By taking this approach, you can apply it to Scrum, RAD, Agile or any waterfall flavour equally well.

1. Identifying the organisational goal the commitment, timeframe and budget.
2.  Identifying a technical solution that meets that goal within constraints
3. Delivering the technical solution within agreed quality constraints.
4. Delivering on the corporate goals.

In the best implementations, step one receives lip service in the “Feasibility” stage and step four is hardly ever taken into account. In the worst implementations two and three are poorly implemented with substandard testing and little documentation.

Step three regularly suffers from confusion between user and investor with the real end user rarely beimg engaged meaningfully.

It doesn’t have to be like this and it most certainly shouldn’t, but the reality is evident everywhere.

Agile variants are all created equal and simply marketing ploys

SCRUM, Agile and DSDM all tackle the business perspective and place it at the centre of the project lifecycle, they incorporate testing and UAT into the development process and they place emphasis on communication at all stages of the process. Scrum is seen to go a little further in considering the user, though the others don’t exclude it.

Regardless of which flavour you choose, you can tailor it precisely to your project’s needs and the idea that one flavour is better than another is almost to suggest that someone who can master software engineering needs to throw out his book and buy  a different one to add a little more emphasis on the customer etc.

Other concepts associated with Agile methods, such as working in teams of two and holding scrum meetings are all engineering technicques that owe no alegiances to any methodology and should be in everyone’s toolbox.

Can agile be scaled?

There is quite a bit being written recently about scaling agile. I have to be honest in saying that I have not read any of it and don’t have any plans to do so in the short term, but I have read comments from people I respect and it would seem according to their considered opinions to be predictably bandwagon in its nature i.e. a new opportunity for trainers and consultants to wrap something familiar in new clothes and sell it again.

Just as we discussed the small company culture and large company culture earlier in this article, the software project also changes in its nature when the team, or the complexity grows and it is inevitable that more layers of management have to be applied and more prescriptive processes implemented and before you know it you will have waterfall regardless of what you call it.

Large systems can and should be broken down into smaller ones and some, or even many of these can be developed using agile methods, but the overall project needs a more planned and less dynamic approach both from an organisational and an engineering perspective. There is no silver bullet and no  substitute for common sense and communication.

The answer is simple, follow the simple four stages and use whatever methodology makes it easiest to achieve each goal and you won’t go wrong.