Putting agile in perspective.

This article discusses an important aspect of agile that is rarely figures in the big debate, i.e. where agile fits in the great scheme of things organiation wise. It also examines breifly  the concerns that are shared with general corporate activity.

It sheds light on  why agile came about and regularly returns to the fore only to disappear again and most of all it puts the whole cult aspect of agile under the microscope arriving at some perhaps unexpected conclusions.

Conclusion

Some people are overawed by the challenge of software development to the extent that they rely entirely  on a comforting framework to lean against, while others use their big picture perspective to manipulate the marketplace selling  their training, books and consultancy services under one disguise after another, while contributing little or nothing to the profession.

In reality there are two major challenges facing all software endeavours. The first of these is the challenge of dumning down science and motivating people to ahieve some sort of useful marriage. The scope is beyind this article but siffice to say that the internet arrived at least 25 years after it could have and is still years behind where it could be while people catch on to the possibilities. The antithesis of Moore’s Law  if you like.

The second challenge facing software projects could be summed up as defining an agreed goal and delivering on it.
in meeting that challenge the professional will follow the one very simple Software Development Lifecycle (SDLC) regardless of what methodology or framework he/she initially elected. That lifecycle can be loosely described as:

  1. Identifying the organisational goal the commitment, timeframe and budget.
  2.  Identifying a technical solution that meets that requirement within agreed constraints.
  3. Delivering the technical solution within agreed quality constraints.
  4. Delivering on the corporate goals.

The key to choosing and using methodology is to understand the challenges thrown up by a specific project and address them correctly. These challenges will be predominantly driven by corporate culture, process complexity, level of innovation, technical competence and budget constraints.

Ultimately it matters little whether you begin by creating detailed plans and evaluating them technically, or start building storyboards and prototypes. Likewise it matters not a jot whether you build it in time boxes, or spend two years coding before shipping the lot to outsourced testers, both of these approaches will have to meet all four objectives before the project is completed successfully.

There are however, certain projects that simply won’t fit a particular approach. One useful example is that you wouldn’t be advised to take an innovative prototyping and experimenting approach to air traffic control systems for example, as the consequences of error in the “trial and error “ paradigm are well known and best avoided.

The reason for the re-emergence of agile as an SDLC as opposed to a methodology is that it includes items one and four of our basic lifecycle in a single project.

Outside of Agile it is very rare for items one and four to be given very much more than lip service.  Business people are generally untrained and ill-equipped to rub shoulders with the technical aspects of this work and are understandably very frightened to make commitments and take responsibility for something they don’t understand and can’t control.  IT people are rarely equipped with the business knowledge and people skills to pull it off even if they were likely to be taken seruously by the busienss. Hence the ostrich syndrome prevails more often than not.

Agile breaks this deadlock by placing empowered people with all the skills they need and sufficient budget into a single team and telling them to make decisions and get on with it.

 

Agile is to the SDLC what a project is to the enterprise

“Agile places empowered people with all the skills they need and sufficient budget into a single team and tells them to make decisions and get on with it.”

Why does this statement seem radical to some people? Surely we have sufficient grasp of management in the 21st century that we can define a task and get it done with a level of motivation and commitment within just about any corporate body.  Well it seems not.

It is well documented and understood that once an organisation grows to the point where the founder can no longer be involved in every decision, there begins a phase that either destroys it, or morphs it into a modern political and mostly dysfunctional organisation that is driven by policies, processes and politics,  with little identity and few motivated individuals willing or able to chase opportunities or to excel.
The winning strategy to survive and prosper in this corporate world is based on being out of sight and below the parapets at all times except miraculously just after a victory and of course bringing apples to the teacher. Excellence is rarely sufficient and often unnoticed.
As these organisations grow within their specialised field, they substitute predictability, buying power and economies of scale for individual efficiency and innovation and they can survive and prosper despite the negative, but predictable performance levels produced by process heavy environments lacking in initiative, or motivation.
The problems surface when you try to use that same team with the same culture to do something different.  Not only are you asking them to step out of their comfortable process, but a successful result will often bring with it an unwelcome and frightening change of culture for them and on top of this you are asking them to make decisions and use initiative.  All this in an environment which, through no fault of their own penalises initiative and chops off heads that protrude above the parapet.

This description may sound like a condemnation, but in truth it is just an acceptance of the fact that we are human and we adapt to our environment and that to achieve things we must begin with an honest, or even pessimistic appreciation of that environment and a plan to work within it.

Enter the Project

Although there are projects used in other environments such as construction where they are in fact part of operations, for the most part projects are used as frameworks for dealing with the issues described above. In fact, even when adapted to areas like construction, they still have a key role in abstracting the task at hand from the corporate cultures of client and supplier in order to progress the work at hand in a partnership or joint venture environment.
A typical project will create a governance structure modelled loosely on a company with a senior executive a PM and board that closely resemble the familiar frameworks of CEO, Chairman and board and operate in a similar way for course of the project. This structure releases them from the corporate culture and gives them the the framework and budget to get things done independently.  Just like real companies, they work well when the team are well chosen and pulling together and less so by degrees at other times. Projects without strong corporate support are up against it unless they have a very powerful and committed champion.

Why agile then?

Traditional Projects work well for large undertakings that are sufficiently in-your- face to gain and maintain interest in the boardroom right through to conclusion. Problems get solved and genuine mistakes are not allowed to become bargaining chips.

The trouble is that most projects don’t gain that kind of attention and are peripheral to busy directors and senior executives.  Board members are sometimes not ideally selected, not terribly committed, or even there purely to be kept informed. Occasionay they are hostile.  In these circumstances many projects lose their way for want of decisions, motivation, or intervention against blockers.

This is where the agile paradigm comes into its own.  Instead of creating this large and very formal structure that won’t actually gain critical mass, the key is to choose motivated, knowledgeable and competent people at the right level in the organisation, combine them with high quality technical and support teams and a skilled agile PM while completely empowering the team to make decisions and get the job done while providing access to a senior executive as and when required.

This agile approach returns the project to a small company paradigm with motivated, committed individuals operating with a shared vision, making decisions, communicating meaningfully, pulling together and getting the job done.

The software profession at its best

Surely we have taken software engineering to a point where a reasonably experienced engineer can decide whether something is achievable, give a time-frame and budget and stick reasonably close to it. Again it would seem to be a singularly difficult challenge for the profession.

Since the late fifties, the software industry has learned and developed along a predictable path that is in some ways remarkable and yet in others frustrating.  The return of the same old questions again and again is partly seen as the rejuvenation of boring old ideas for immediate financial gain and this view is not without any truth, but it is also part and parcel of how humans increase their knowledge.

Hegel described the increase of understanding in terms of: Thesis, antithesis ad hypotheses.

In effect he suggested that learning begins with theses, some of this is then rejected (antithesis) and finally, on reflection it emerges for use and further consideration (hypothesis).
Software engineering has developed at a rapid speed through borrowing from hardware engineering, construction and manufacturing, trying these concepts, improving them and trying again. Between 1950 and 2000 the average cost per line of code in use at the US Department of Defence dropped from $10 to $ .000001 as the industry matured. The industry has progressed rapidly in engineering terms, but there’s still a long way to go and especially in terms of interface with the users and investors.

Coding time estimates

In today’s software engineering environment (as opposed to a newly formed team of programmers at random company ltd) the time taken by programmers while programming is roughly as follows:

timebreakdown coding

In fact as a good rule of thumb, it is widely accepted that when a programmer tells you how long a job will take,  you need to multiply that figure by somewhere between 6 and 11 depending on his/her track record.

If this software is to be of any use it must also be documented for the purpose of maintenance and for implementation, training and user support. In a less mature team these figures would be easily multiplied by anything up to a further five.

Communication time estimates

As you build a team the time for team and project communication begins to rise exponentially as the team size grows and if change is frequent it can reach as much as 50% of all effort with alarming speed.

Propensity for change

 

 The cost of changing requirements/features grows rapidly according to how late in the cycle the change occurs.  This change is mostly driven by omissions and errors at the design phase and design or programming errors further along the cycle. Controlling these changes is down to excellent stakeholder communicat

Impact of change in a software project
Impact of change in a software project

ion and rigorous design and verification procedures.

The preceding paragraphs highlight the very topmost concerns in the simplest possible way yet it is clear even from this short commentary that providing even reasonably accurate estimations is extremely challenging and is a whole team exercise.

It is also worth noting that of the four steps defined in our underlying SDLC, the software engineering discipline only covers numbers two and three.
Responsibility for defining the business goals and their parameters as well as responsibility for implementing the system in order to capture those benefits remains an equally challenging proposition and lies entirely with the business.
In a product paradigm, the only difference is that product management must take responsibility for customer consultation and feature design.

Software Development Lifecycles (SDLC) are over prescriptive and misunderstood.

The many incarnations of SDLC will generally expand very quickly into a complex map of steps with interdependencies. With the best intentions, they attempt to tie the practitioner into a very prescriptive set of steps, but in doing so they remove the emphasis on common sense and good communications. For the sake of simplicity, when appraising a project, I tend to simplify the lifecycle to just four goals that can be carried out in any order that serves your specific project and revisited when necessary.  By taking this approach, you can apply it to Scrum, RAD, Agile or any waterfall flavour equally well.

  1. Identifying the organisational goal the commitment, timeframe and budget.
  2.  Identifying a technical solution that meets that goal within constraints
  3. Delivering the technical solution within agreed quality constraints.
  4. Delivering on the corporate goals.

In the best implementations, step one receives lip service in the “Feasibility” stage and step four is hardly ever taken into account. In the worst implementations two and three are poorly implemented with substandard testing and little documentation.

Step three regularly suffers from confusion between user and investor with the real end user rarely beimg engaged meaningfully.

It doesn’t have to be like this and it most certainly shouldn’t, but the reality is evident everywhere.

Agile variants are all created equal and simply marketing ploys

SCRUM, Agile and DSDM all tackle the business perspective and place it at the centre of the project lifecycle, they incorporate testing and UAT into the development process and they place emphasis on communication at all stages of the process. Scrum is seen to go a little further in considering the user, though the others don’t exclude it.

Regardless of which flavour you choose, you can tailor it precisely to your project’s needs and the idea that one flavour is better than another is almost to suggest that someone who can master software engineering needs to throw out his book and buy  a different one to add a little more emphasis on the customer etc.

Other concepts associated with Agile methods, such as working in teams of two and holding scrum meetings are all engineering technicques that owe no alegiances to any methodology and should be in everyone’s toolbox.

Can agile be scaled?

There is quite a bit being written recently about scaling agile. I have to be honest in saying that I have not read any of it and don’t have any plans to do so in the short term, but I have read comments from people I respect and it would seem according to their considered opinions to be predictably bandwagon in its nature i.e. a new opportunity for trainers and consultants to wrap something familiar in new clothes and sell it again.

Just as we discussed the small company culture and large company culture earlier in this article, the software project also changes in its nature when the team, or the complexity grows and it is inevitable that more layers of management have to be applied and more prescriptive processes implemented and before you know it you will have waterfall regardless of what you call it.

Large systems can and should be broken down into smaller ones and some, or even many of these can be developed using agile methods, but the overall project needs a more planned and less dynamic approach both from an organisational and an engineering perspective. There is no silver bullet and no  substitute for common sense and communication.

The answer is simple, follow the simple four stages and use whatever methodology makes it easiest to achieve each goal and you won’t go wrong.

Planning for project managers.

How important is planning?

Planning is critical. Without planning there is little chance that you can every complete your project, let alone complete it on time.

The act of preparing a plan, if done correctly, will uncover the issues and risks, provide the bulk of key data for your estimation efforts, provide a clear view of what resource is needed when and lots more.

It will also help to discover the dependencies that may exist with other projects and activities.
Once prepared, the plan gives the team a better view of how their efforts will come together and points out the need for communication in critical areas.

How important is the plan?

Much less important than the act of planning, but still very important. The reason I say this, is because:

1)  It is rare to be able to complete a first draft plan and then not have to make any adjustments.
Most plans develop as they go along and reach a baseline when well into the project timeline.
The most obvious examples of this are projects that involve investigating a problem designing a solution and then finding suppliers to deliver it.
You can allow extra time at the start in the hope that it is definitely too much , but even then, the chances are that you will end up adjusting your plan.

2) Risks and opportunities are a part of every project plan. Sometimes risks come to fruition and they affect the plan profoundly, sometimes opportunities come along that are either too good to miss and causes change of requirements, or  reveal a cheaper, or faster way to get it completed.

In either case, the initial plan will have changed. To live in denial of this as many commentators on project management still do, is a huge mistake and will always be counterproductive.

E.G. If you become so obsessed with meeting a specific date that you are prepared to pare away key features, there is a strong likelihood that the project will fail entirely. The answer is to treat the plan as a guideline and treat planning as an ongoing task.

I recently had this same discussion with a group of seasoned Venture Capitalists and their view was this:
They would never dream of setting out without a plan, but once they had it in place, they may as well tear it up, because it had already delivered most of it’s value and from here on in, it was more about managing the risks and spotting the opportunities.
They were unanimous in the view that to slavishly stick to that plan would be suicide virtually every time.

Starting a new venture is much more volatile than starting a project, but it is also higher pressure with greater risks and a lot more to lose.  They don’t actually tear up the plan of course, they adjust it and maintain it, but the discussion served to get their point across that management is about managing and adapting on a daily basis, not stubbornly following a plan just to prove that you were right.  Read Maseena Zeigler on forbes about this subject
Managing a project should be driven by the business case in the same way that a new venture is driven by reaching a profitable trading position at some point.

Critical also is the acceptance that like a start-up, many good projects are based on a strong hunch and a bit of a gamble for a wothwhile prize. This is enterprise nd without there would be no paydays.  project managers haven’t earned indemnity from this either. 

Features – Deadlines -Budgets – ROI

Here’s where it all happens.  A project will have a goal and that goal will usually be a financial one though success is not always measured in financial terms.
For the sake of this example I will assume that the project is a cost cutting exercise with a specific financial aim. Right at the beginning, the same ground rules should have been laid down such as;

  • What saving are we aiming for?
  • What investment are we willing to make?
  • What is the maximum our budget can rise to, or the minimum our savings can drop to.?

This latter question is best answered in terms of ROI and in fact, in most commercial organisations the answer will be worked out on the basis of how much ROI exceeds  “Cost of Capital” in order to qualify the project as “best use of capital”. The critical thing is that it is agreed in advanced and set up as the target.

Once this understanding is in place and the variances have been explored and allowed for, the business case should clearly show the expected returns and the tolerances that are allowable.

The project now has an ambitious goal (maximum realistic returns) and realistic allowable variances. The importance of these variances is not to make the project management team relax, but to assure the sponsors that their investment is relatively safe. Projects set up this way rarely fail.

MSF, the Microsoft flavour of project management introduces a very useful concept known as the project triangle. It is a simple triangle with the three corners being occupied by Features – Resources – Time.project triangle

The significance of the triangle will be obvious to any mathematicians reading in that only one corner of a triangle can be fixed unless you want all three to be immovable.
This is why the prioritisation of concerns is a valuable part of stakeholder alignment. If stakeholders are allowed to follow their natural instincts and demand that all three corners are fixed (two fixed corners means the same thing) then there is no room for the project management team to steer the project out of trouble.

A realistically aligned project will choose one of  Time, Features or Resources to set in stone and the other two will remain free to move.  This way, if time is fixed, then a PM can choose between increasing resources, or decreasing features in order to hit the target (ROI). The decision making process can also be agreed in advance.

This example is one of the simpler ones, but it is indicative of the ills that beset many projects right at the beginning and rob them of any real chance of succeeding, or being seen to succeed.

Defining the scope/budget/resources

Once again, with the ROI in mind and the statement in the business case that based on initial studies, there is a strong likelihood of success, the task now arises to define in more detail the features required to deliver the expected benefits.

Having defined the features, accurate costing has to be worked out on the basis of supplier estimates, internal efforts and other costs, with adjustments for risk, all of which will rely on your emerging plan.
Another sanity check at the end of this planning phase should check whether  the cost and time estimates are still within the constraints initially set for the project and whether it has a healthy amount of slack remaining to see it through to a likely successful completion.

This exercise of working up plans from draft to more detail as the work progresses from an outline business case through to a detailed contract with suppliers and a detailed plan of implementation with all the appropriate slack allowances  and a rigorous risk assessment will often take up half of the lifecycle of the project, or even more and some projects will be abandoned when it becomes obvious that there is little likelihood of success.

The number of “stage gates” (sanity checks) should be agreed at the start on the basis of how well known the territory is and how volatile the estimation is likely to be.

The illusion that a project board can put some dates (when we’d like it done) and amounts (how much we’d like to spend) on an A4 sheet at the beginning of the exercise and that perhaps a year or more later, a project team will deliver exactly what was asked for on that sheet, within exactly that amount and on that date is really a surprising error of judgement, but it still happens and contributes strongly to the list of project failures that appear on the Standish report and other investigations.

How to go about project planning

Planning should always be done by starting at a very high and general level, involving experienced big picture thinkers and applying a sanity check before then drilling down not too far to the next level  detaiil to repeat the exercise.
Planning should always resist the temptation to go into great depth in one specific area while remaining at a high level on others with one exception.

If there are unknowns, e.g new concepts that might not work, then proof of concept should be carried out as soon as possible to avoid having to abandon the project or change tack after a great deal of money gas already been spent

Plans should not be in extreme detail a long way in advance, because the likelihood is that when that time draws closer you will find yourself redrawing them and the effort has been wasted.  As planning continues to drill down, a plan should be retained for each level of detail and when the detail highlights an error in the high level plan, as it frequently will, then the high level plan should be adjusted.

The commonest method of estimation to begin with is to break the project down into products.

These products can in turn be broken down further until they reach a level whereby they can be more easily estimated and planned and later assigned to teams.

Out of this comes a product flow diagram that describes the order , if any, in which these products must be completed in order to take account of interdependencies.

Calculating critical paths ( the longest path you can follow) through the products and then amongst the products, the project manager can get a much closer view of the true schedule of the project.

Using PERT to make a high level allowance for uncertainty adds a further level of sanity check to the emerging plan.

Some of the issues that commonly affect project plans drawn by the unwary include:

1. The calendar is not taken into account when calculating for tasks in the plan, e.g seasonal breaks, Summer holidays and other disruptions that happen every year.  Key personnel  disappear and dependencies become critical.
Make sure you discuss each team members responsibilities and schedule with them and get agreements.

2. Suppliers work to their own schedules and regardless of what they agree,  they will place commercial concerns first and may not follow your plan.
Ask them for their plan and question to satisfy yourself that it is thought through. If you  lead, or take part in the contract negotiations, try to place some extra responsibility on them to deliver on time, or warn you in advance.

3. Estimates from technical people are accepted at face value and in reality they are vastly underestimated 80% of the time.
Get second and third opinions, look at records of old projects and as a minimum double the estimates from the best people and use factors as high as five for others.

4.  There’s gaps between what the supplier delivers and what the internal team have allowed for.
e.g. Inexperienced people might assume that a system can arrive, be plugged in and start testing.
Clear acceptance criteria may not exist and there may be problems agreeing acceptance.
Cut over from an old system to a new one may not be catered for.
This list can grow quickly. If you are not an experienced systems person make sure that there is one in charge of this part of the plan.

5. There may be several suppliers and communication between them may be less than ideal. Often this situation leads to gaps where nobody is responsible and the work grinds to a halt, or even enters dispute , or litigation.

Make sure you hold joint planning meetings and get sign-off to theses joint plans

6. Beware Johari’s window. What you don’t know you know is a terrible waste, so consult and consult again. What you don’t know you don’t know will come back to haunt you, so involve everybody in risk management sessions and try to be ahead of the game, have sufficient slack and keep stakeholders informed of the true position.

7. Over ambitious, or over confident plans create a sense of expectation amongst stakeholders that changes to discontent when the plan slips. Perfectly good projects are often deemed poor projects as a result of this very mistake.

Take the time to estimate risk realistically and maintain realistic slack for the high risk areas of the project.

So in summary:

1. Planning is critical and underpins everything, but it is an ongoing everyday task, not a game of snakes and ladders.

2. Managing projects is primarily about handling and working with uncertainty to remain within agreed, acceptable boundaries, not shooting a silver bullet at a precise point.

3. Planning for well known items is easy, beware what you don’t know, this is where you will get caught out.

4. Risk and opportunity go hand in hand, avoiding risk is no more important than spotting opportunity , but it requires an open mind and a fluid approach to planning.

5. Start with an achievable goal, make sure it is well understood and shared  and keep it in front of mind at all times.