UAT – me, how do I do that?

Previous installments

IT investment for the smaller business
Why the Smaller business holds all the cards
Where has alll the money gone
Requirements gathering the first big mistake
Contract negotiation and on we go

Well here we are, though dragging slightly due to the unexpected holdup when everyone was so inconsiderate as to take a long break in the Summer and get all relaxed. It seems like a very short time despite the months that have passed, but we are now nearing a milestone when the vendor delivers the systems to us and according to both the vendor’s project manager and the IT manager, we are expected to do something called UAT.  Of course it’s not written in law anywhere, but to our project manager, it may as well be because he is following the conversation two sentences behind and trying not to look too foolish.  The vendor is still driving the agenda alone and the IT manager is still sulking, but gaining in confidence as the project approaches his territory.

The problem with COTS purchases and traditional testing methods is just that.

Well, it’s just that the methods were designed for a world where teams of engineers spent months writing code and then began to stabilise it and introduce it slowly into the business environment until it was a stable release and a good match for the tightly defined requirements.  At least that was the theory, there were usually problems about interpretation of the requirements and about how the outcome was actually achieved, but at least the bits did fit together and they followed a well established pattern. Vmodel was and still is the standard.

 That was a simple theory, you had to created business requirements, the analysts translated this into designs, the developers translated that into systems, then it was refined to make it usable and fill the gaps in interpretation of business requirements.

Logically therefore, you tested the business requirements with stakeholders and got it agreed, then the designs with business analysts to make sure it was properly interpreted, then the system to make sure it fitted the design. Along the way, there had to be technical testing to make sure it could run on the clients environment correctly and then a bit of testing to make sure it worked well enough for testers to commit weeks or months of their time.
Finally you arrived at the point where you had a working system that had no major bugs and appeared to meet the needs of users and it was time to let them loose on it.  That was UAT.

It usually flushed out little bugs that had been missed, it also flushed out poorly interpreted requirements that just didn’t work in a live environment. Additionally, it served as a way to win over key users and pre-empt any likely resistance to the process change aspect.

For a simple explanation, that has taken a bit of effort and probably lost me some readers, but it’s not a simple business.

So how does our project manager apply this to a piece of kit arriving on a CD with a few minor customisations and integration endpoints?
Maybe not the million dollar question, at least not every time, but an expensive one often.

Our friend has no detailed requirements to work to, because the initial ones were not adapted, but instead, the sales guy sold an existing package and said it will meet the requirements you described. If our friend hires a tester, he will need to go to the vendor’s and ask them to help him list exactly what this system is expected to do for the client in order that he can test it.
Were he to do that, alarming as it may sound to many readers, at least a start would be made on understanding what has been agreed, what is expected by whom and what the likelihood is of all these stakeholders being in the same book let alone page.
Now I am not a sadist, despite what you may be thinking and I have no intention of bringing up the business case and the business goals at this point, our poor friend is in enough trouble already.

The IT manager is now beginning to ask awkward questions and making apparently unreasonable demands about some sort of complicated and time consuming testing before he lets us install our system on “his” environment.  Is he out to get us? He never really was supportive anyhow, because he felt he should be in charge.

So where have we got so far?

We identified the need for change and made a business case for investment of the businesses capital based on some modest assumptions sound process change and cost reduction due to efficiency and we got the go ahead to do it.  We have not tested these assumptions against the new system being proposed and we have no idea whether any of the goals will really be met other than a stirring presentation and “good feel” from the vendor’s sales rep.

We found a supplier that we felt we could work with and agreed a contract, we know of course that that contract was tipped 90% in favour of the supplier, but we hoping that all will end well.

We are now approaching eminent delivery just a little behind schedule, but nothing to worry about and we are very confused about the communications coming from the vendor’s team. We are expected to complete “UAT” in three weeks and then sign off acceptance and pay the final instalment for services and we will begin to pay the support and other fees immediately. This has a very final and serious ring to it.

We are gathering up a few people and arranging a room where they can sit and “play with” this system for a few weeks. Then hopefully all is well and we are done.

We have met our milestones so far, the budget is at or below forecast and the CEO is very impressed with us, let’ s hope these IT guys don’t spoil the party.

Next:

Whose fault is it anyhow ?

 

Reblog this post [with Zemanta]

Contract negotiation and on we go

Previous installments

IT investment for the smaller business
Why the Smaller business holds all the cards
Where has alll the money gone
Requirements gathering the first big mistake

A shock for the project manager

In the last instalment we discussed the way in which a vendor had been chosen that near total disregard for the goals of the project in the course of making that decision. Now we will look at what happened after the decision was made.
The next agenda item directly after awarding the contract was to negotiate and agree details and on this occasion it was lead by the vendor’s legal team and the sales rep.  The atmosphere was very warm and jovial and nothing was too much trouble in the vendor’s quest to make the new customer happy.

So far so good

The contract was agreed just after lunch on day two and everyone went away with a sense of achievement.
Deliverables were carefully defined and the acceptance criteria was very forgiving. It would have been clear to any pro that there was a lot of stuff not mentioned that would need to be done and paid for, but it was not brought up.  The bulk of the cost was loaded up front with a year’s licenses to be paid for immediately and support to begin charging on contracted release day.
There was an SLA, but it gave no guarantees of times to fix issues, only times to address them. The rate for professional services was a little steep at £1400 a day but it seemed academic.
The real problem with this contract was that it never referenced the now academic requirements documentation (In any case it was pretty useless) and
the contract revolves entirely around delivery of a certain feature set as described by the vendor in features language, not in business need language.
Success criteria is defined as delivering this feature set by a given date and to a given cost
At this point we have two teams involved; a project team that is now providing services to the vendor to get the job done and a vendor with the goal of delivering the described features and getting paid.   Our initial problem of lack of experience and know-how in the sales and marketing teams has never been addressed and is not on anybodies radar and there is nothing anywhere, either spoken or written to say that delivery of this feature set will solve the defined problems I.E. the business case has been forgotten. In fact the thing that really swayed the decion on which vendor to use was a very sexy system that lets  the user record telephone conversations and the hardware to make this work pushed up the cost.  
Here’s the shock for the project manager.
The sales rep is now cosy with another potential client and the project manager is now dealing with the professional services team tasked with delivering the project to  tight budgets.   Now he’s dealing with a real project manager and a hard nosed business man who survives by taking no prisoners and earns bonus by growing the contract value through extras. He is a past master at dealing out the rope and charging to reel it back in and he’s seen all the snivelling before. He is the essence of tact and diplomacy but his hand is firmly in the project managers pocket.

The roller coaster

Within two weeks of contract completion, the mood has turned frostier and it is clear that this is not precisely what the project manager had expected and hoped for.  Everyone is now locked into a roller coaster of milestones and budgets and the reason for it all is completely forgotten. That was handled in phase one, wasn’t it?  You get on in the corporate world by meeting milestones, not by delivering results and certainly not by making waves.

We are where we are

We are now on ride that won’t slow down until this thing, whatever it is going to look like, is delivered and we can start the job of making it work and getting people to use it so we don’t lose face.
Nobody who suspects the truth, if such a person exists, has the seniority or the courage to step out of the line shout horses@”!* and nobody with the authority or the presence to do something has any idea that things are not what they seem.

 In the next installment we will look at the the end game

About the author

Requirements gathering the first big mistake.

Previous installments:
IT investment for the smaller business
Why the Smaller business holds all the cards
Where has alll the money gone
 Many seasoned project managers will start off a project by sending out his business analysis team to carry out requirement gathering.
Sometimes this is the right thing to do, in particular you should do this when you have a successful proven process that you want to automate using IT.  Requirement gathering then educates the IT people responsible for system design (the features) on what they need to achieve.
This is a different scenario, just think about it for a minute.  If we had the business case open and we had a succinct explanation of our goals and how success will be measured, would it not say things “like failing processes”, “lack of experience and knowhow” and would there not be talk of providing a new and better way and retraining people?

Here’s what actually happened.   
A Business analyst was hired to gather requirements and she went about learning how the business works and mapping all the tasks that people do every day. She was very thorough and presented it all back in neat UML diagrams with notes and she created a data catalogue to back this up.  An up and coming manager from the business then was appointed to find a supplier and deliver a system. He was instructed to report with final costs from shortlisted suppliers.

Off he went and began to invite various CRM vendors over to give presentations and answer questions.   They were given copies of the Business analysis in advance which some read and few paid much attention to.  Fairly quickly it became evident that this was a bit technical and he would have to rely on the vendors to keep him out of trouble and he quickly got it down to two people that he felt he could work with and asked them for quotes.  He had by now built relationships with these guys and they were advising him. The IT manager was also sitting on occasional meetings to make sure things were to his satisfaction.

He ruled out several vendors early on because their technology was not compatible.  He was a bit miffed that he was not in charge of this instead of someone who knew nothing about technology. It never crossed his mind that he knew nothing about business. He insisted on writing a features grid based on the business analysis and asking the suppliers to state whether they met all the requirements.

Eventually they chose a vendor that the project manager felt comfortable working with and that answered all the questions for the IT manager.

Perhaps this is a good place to finish this instalment and focus the attention on where we have got to from where we started.
We began with well defined business problems that required to be addressed urgently and which included lack of experience and know-how in the sales and marketing teams.  We ended up with a supplier chosen on the basis of the IT manager’s priorities and the Project managers relationship with the sales rep.
Par for the course.
At this point the budget has a debit side exceeding £30k and that takes no account of internal staff spending considerable time away from the day job.

Next week we will look at the contracts negotiation stage
  About the author