Image result for bot wikipedia

I wont be long-winded about this, I’ll discuss it via email with anyone who is interested, but I’ll break with my usual mode and come straight to the point.
A great many people who know little at all of machine learning  and even less about people and many more who are simply  oblivious to the potential consequences of their words are talking about the miraculous things we can expect from Machine learning.

What is ML in a nutshell?
Academics break ML into two modes:  Supervised and Unsupervised.
In the case of the former we give the machine a large corpora of content and ask it to decide what will happen next, or to find other similar instances. A translation service for example  begins this way and learns after a while to translate without help.
In the latter case, we give it a body of content and ask what it thinks of that.. Google search is an example of this approach and it simply makes sense of what it finds.

Often we give it a few hints like “Classify this for me and establish links” as in Google search. This would be a “Classification problem”. We might on the other hand ask it to read the racing papers and decide who will win the four o’clock today. This would be a “Regression problem” because we are asking it to look at the past and predict the future. Yes all of this is highly condensed as promised, if you are an expert you don’t need my explanations.
Understanding what the customer will want next year, predicting the weather, finding Oil under the sea, predicting tumours, the challenges are endless and the rewards enormous.

What is the loop of self-destruction?
The loop happens when, thanks to social media, a good, but no a sole example, the machine begins to make judgements that influence the data and then discover exactly what it predicted.

As with humans this will give it the machine equivalent of a big head and possibly some citations and will lead to even greater confidence and fewer checks and before anybody spots it, it I all too late.
If any Movie producers out there are stuck for an idea, I am available to help with the plot. Here is a simple example we are all aware of:
Joe Gel, and Josephine Lotion our dear friends, represents an enormous body of intelligent and informed people who spend most of their waking hours  checking back with their phones for reassurance. Joe searches Google for Tom Raspberry, his favourite politician and receives a huge list of pages. The ML in google notes his interests and begins sending him dozens of articles about Tom Raspberry, what he says and does and what people say about him. Unwittingly Our pal Joe has become astonished by the fact the whole world seems obsessed with Tom R and realises subconsciously how important to is be aware of Tom R. He begins to tweet and have the odd Facebook conversation about something he read. Immediately the ML in Facebook and the one in Twitter hone in his apparent obsession with Tom R and all begin to bombard him with content and introduce him to thousands of people with the same problem. Poor Joe.

Now our Machine does a Recce to see what are people talking about and it discovers that millions are talking and reading about Tom raspberry and concludes that tis is the way to keep the customer happy so it ups its game and heightens the emphasis. It also confidently announces that Tom R will undoubtedly be unstoppable in the forthcoming election.

Joe and Josephine realise the importance of not standing in the way of a social crowd and are not about t be shunned and subconsciously they begin to take more interest in the positive stories about Tom which now triggers the Machine to filter their feeds and search results and friend recommendations etc more toward the positive . You don’t need me to finish the plot. There is only one way this is going. Imagine if the secret services relied on this kind of information to brief their bosses. But they do, don’t they.

You may well think, as I do , that despite the  shear “wrongness” of rigging democracy, whether by design or accident, it matters little who is elected anyhow. In that case imagine the same scenario when the machine turns its hand to guiding change in a government department or a large business , or guiding product development or even finding the cure for cancer. If you would like to see many better examples with a strong scientific analysis, check out Weapons of Math destruction.

One wonderfully simple yet highly destructive outcome of ML that I have seen up close is the  call centre  automated system that recognises your telephone number, calculates your value as a customer and decides if you will be answered, how long you will have to wait and whether you get to speak to somebody skillful.  Just to update my card details for a £20 a month hosting service, I had 11 hours of my time wasted, had my service disrupted and was threatened by a bot with £150 fine to put the service back on.
I hate to disappoint you, but if you have ever had an IM conversation with a patient lady on the support portal “That was no lady” nor was it my wife, that was a distant cousin of Cortana.
If she did not know the answer, or more likely the question, you were never going to be served.
If you are wondering what might happen to your pension, your job and your home if these guys get involved in stock trading, well take a look here  According to a 2014 report, sixty to seventy per cent of price changes are driven not by new information from the real world but by “self-generated activities”.

It’s not all negative by any means. I actually do use ML to predict the winners of tomorrows racing with a consistent level of profit. When I get it wrong, usually after a late night of programming with insufficient testing, my winnings disappear very quickly into someone else’s pocket and I sit up and take notice.
I sincerely hope that someone starts sitting up and taking notice soon  of the impact of poorly programmed Bots that are already beginning to increase risk for the most powerful nations on earth.

Don’t forget about innovation

The first part of any project involves defining  the problem, deciding where to look for the solution and how to proceed with the search and finally defining the solution, validating it and getting agreement from stakeholders.

Now the nature of Technology is such that few of us are aware of what is possible and even fewer are able to see the impacts of these suggested solutions over and above the promised outcome.

Not only are few of us equipped to access the best solutions, but even fewer are able to recognise when we have a problem.  In technology speak a problem is closer in meaning to a mathematics problem , it doesn’t necessarily cause that irritating pain that our marketing colleagues like to focus on.

E.G.  Let’s say chief zongawonga is worried that with 19 more children due this spring, he won’t  be able to catch enough fish.  His bright young progeny identifies the problem and suggests metal arrow heads that are more effective and mean they can quickly make extra spears so everyone can join in. That represents a problem known and tackled.
However, Zongawonga doesn’t know that monofilament nets are cheaper than arrowheads and one child can feed the whole tribe with one net.  Until he becomes aware of the nets, he won’t know he has a problem, or until his wives start leaving for the easy life with his neighbour who doesn’t expect them to fish.

The process involved in definition of problems and solutions differs not at all from the age old problem of effectively searching a global mountain of unstructured content as described below.

First you have to arrive at some fundamental conclusions about the problem, the possible solutions and how and where to go looking. Consider this example and then have a read through the innovative solutions put forward by Zyra and see if you don’t begin to question the stuffy, stuck in the mud methods of innovation and improvement that have become embedded in most modern businesses.

“just what are you looking for, anyway?”

  •  A known needle in a known haystack
  • A known needle in an unknown haystack
  • An unknown needle in an unknown haystack
  • Any needle in a haystack
  • The sharpest needle in a haystack
  • Most of the sharpest needles in a haystack
  • All the needles in a haystack
  • Affirmation of no needles in the haystack
  • Things like needles in any haystack
  • Let me know whenever a new needle shows up
  • Where are the haystacks?
  • Needles, haystacks — whatever.

http://www.zyra.org.uk/needle-haystack.htm

 

The answers you give to the questions above will have a profound effect on how you approach the project, how you define success and your likelihood of succeeding.
Furthermore, whether you are in charge of developing  market leading products, or keeping your company  at the cutting edge, taking a little time out to consider these questions and address them  innovatively will take your performance to the next level.

What to watch out for when researching and testing journeys

Previously:

Why you need to pay attention to customer experience

What to watch out for when researching and testing journeys

 

1. Last Click  can be a costly mistake

The customer journey does not begin with the last click before they visited your ecommerce site. No assumption could be more damaging than this one.

Here’s an example of what happens before “LastClick” (Uses App to order)

 UJ1

You need to take off that blindfold and clean your glasses if you’ve been ignoring this part of the journey.
Once you have got ahead, next you will need to criss cross this journey with the interactions with your competitors if you want to stay ahead for any length of time.  And don’t forget all those potential customers who have been influenced by this on-line saga.

For an interesting interactive tool that can be filtered by industry classification to get a glimpse of how last click relates to the other channels and influencers, check out the ink below

http://www.thinkwithgoogle.com/tools/customer-journey-to-online-purchase.html

 

2. Its not what they say, or what they do, or even what they say compared to what they do, but it’s what they tell their friends that is interesting.

If you have spent any time working with professional market research you probably noticed that people tell you what think you want to hear, what they think makes them sound clever, or mostly what they believe the group will be impressed with.

It takes a great deal of skill to capture useful insight from market research and to make large investments on the basis of just a market research report, however good, is a fatally risky thing to do.
People invariably do entirely different things to what they expected they would do in a particular situation and even from what they planned to do.  It can be very difficult to explain why you did this, you can read the underlying Social Psychology theories and decide whether to believe them, or not, but it wont change the facts at all.
People are more driven by their acceptance in, or leadership of groups than any other thing and hence, once they announce to the group what they are going to do,  it is more likely that they will do it than not.  Hence the most valuable insight into what someone will do is what they are wittering about in social media.
Be warned of course, just like other forms of research, this is one more voice to add to the conversation, not the single source of truth.
Just in case you are thinking it, most people ask this , there is a big difference between what people say in idle chatter to their friends and what they tell you in focus groups. The latter is not populated with their peers and it ends when they leave.

 

3.    The journey does not end at purchase, or anywhere close to it.

If you have been following the series so far, you will remember that in paragraph 5 of the last section I presented some hypothesis about the power of referral from your customers.  This is a very valid argument indeed for being switched on to the journey after purchase. In many sectors of course ,there is an ongoing relationship with the brand.

First of all the product may be one that is repeatedly used over time, each repeated use will therefore be another interaction with the brand and the quality of that interaction does count. Additionally there will be the difficult period when customers are learning to use your product and later there may be issues that require active support.

If you take the view that once they have purchased and you have their cash, there is little motivation to spend more money on them, then you are greatly mistaken, because even if they never plan to make another purchase of your product, they will talk to everyone they know and their influence will cost you dearly. Put together enough of these negative voices and it will put you out of business.

A far more constructive way of thinking is to ask yourself what up-sells, or cross-sells you might be able to introduce as a result of providing high quality personal support. If you don’t have other products, this may be the catalyst you needed to re-think your brand and product strategy

4.  Pay special attention to understanding Critical touch points (Moments of truth)

A user journey can be and usually is very complex, even when we only represent the main points. Even this however, is a potentially misleading picture. Thee are always Moments Of Truth when you win or lose on the tiniest of margins and these are the ones you must identify and focus your closest attention on.

For example: When the customer finally returns to your store feeling really happy with her decision, clicks buy and you don’t have her size in stock. That’s not a good situation.

When she clicks buy and gets a spinning thingy that goes on and on. Not good.

When she buys and although being a regular customer she is asked to enter a lot of information yet again despite being on a sunny bench with her HTC One and her longest nail extensions. You are really pushing your luck here.

It’s not all about clicking buy, it could be when she decides to see what people are saying about you and the first thing she finds is a blog by an irate customer of yours that you didn’t even bother to respond to let alone apologise for. The writing is on the wall.

These moments of truth may differ for different customer segments and personas, so you really need to know this and it may even differ at different times of day or seasons or with different devices.
For cheap flights customers I would say that, certainly for me it is “Paying the bill”,” Leaving on time”, “Arriving on time”.  The other stuff in this case is far less important and if it can only be done at the cost of these three, I hope they just ignore it.

“Learn what the Moments of truth are for your customers and focus a lot of your attention on getting them right, it will make a huge impact on your top and bottom lines.”

ZMOT-GRAPHIC1

According to Google, there’s a Zero Moment of  Truth.

“that moment when you grab your laptop, mobile phone or some other wired device and start learning about a product or service you’re thinking about trying, or buying.”

When I talked previously about taking a hit to let customers realise how badly they want your product, this was ZMOT.
When Henry Ford gave them a smelly noisy Iron horse and changed their lives, that too was ZMOT.  For you and I, winning at ZMOT means deeply understanding our customers habits and preferences so that we can be there at the moment when the inspirtaion hits them and showing the the right messages to take them to the next level in the journey.

1MOT

In bricks and mortar terminology, this is nothing more or less than “First Impressions”.  Until Google gives you the time machine, you can’t go back and redo first impressions.
We spoke in the last instalment about how People see what they expect to see and feel what they expect to feel,  well this is ground zero. Here is where you set that expectation and it had better be good.

2MOT

This is when they open that box and your shiny product falls on their toe and puts them in hospital, or they struggle with one point white print instructions on grey papaer in broken Korean. You may well have thier money in the bank right now, but dont fool yourself about the importance of getting this right.

3MOT

Is the remebered sensations of using your product and how they compare to the expectation.  Wow that requires some thought.
Its not enough to have a great product, but it should be at least up to the expectation. Not only do they need to like the product, but they need to trust you to give them an even better one when your competitor moves the goal posts with a new offering, otherwise you’ll be saying an early farewell. Don’t forget that it is not just your product, but everything your business does in the public eye as well as how you answer your phones, or not and help them when they have a problem.

4MOT

This is when they talk about you to their friends and social media contacts. This is one you absolutely must win

Why you need to pay attention to customer experience

Why you need to pay attention to customer experience

Next        What to watch out for when researching and testing journeys

 

The chicken and egg question always fascinated me. When it comes to business models I find the same conundrum with customers and profits. Michael Porter once said that the purpose of a business is “to create value for customers”. Although we all assume it was inferred in there, he never bothered to mention profits.
The reality every business faces however it that creating value comes first and monetization follows.

1. Compare the debacle of the great Thatcherite privatisations to the often maligned success story of dot com.
In the UK we have a raft of privatised utilities who still have not “got it”, they still think in terms of Oligopoly, force, bullying, price rigging. They think and act like tax collectors. The total innovation from all of them over two decades could be written on the back of a credit card along with a full list of their happy loyal customers.
Amazon, ebay, Paypal, Google and many more have on the other hand built world beating businesses on the back of profitless customer satisfaction and only now are monetising these business models. They operate at P/Es up to 500 and have no shortage of investors.

The message is clear, the customer is king and until you can demonstrate value to them you don’t have a business model.
“ Sooner or later regardless how much cash you have stashed away, you will learn to create value for customers or fail.” We even see this law apply itself to dictatorships.

2. What is customer value and how can you create it?
The biggest possible blunder any business can make is to quantify customer value in terms of product features. I cringe when I see these neat spreadsheets listing product x competitior1, competitor2 etc and how well they score on each (in the marketing trainees opinion).
Customers buy an experience, even hard nosed corporate customers. That begins with the interaction with “People” in the supplier side, or “friendly” and human like ecommerce site and carries right through to anticipating delivery, opening the package, using it for the first time, bragging to friends, interacting with support and many more. Many of these are remarkably powerful influencers and even though supported at times by product features, most of the time they are a separate source of value, or indeed antagonism.
Next we return to the chicken and the egg.

3. Does customer experience exist without customer value and who foots the bill?
The problem here is thus: If you ask the customer how much extra they would pay for their phone to float up out of the box on a mechanism with a Jingle playing, be fully charged, sense the old phone and offer to copy the contacts and messages etc in a sweet voice, accept a voice answer. The customer might well offer a price that made this simply not feasible. However, when that same customer experiences it once, the likelihood is that she won’t want to be without it and when she hears her friends talking glowingly about it, it becomes a must-have at almost any price. Soon it is talked about and develops a cult status and then we have a brand value to take into account. That’s a whole new ball game.
I’m not suggesting we deliver high quality customer experience at all costs, I’m simply saying that you must understand the true value and what people do is far more revealing than what they say.

The point I’m making here is that sometimes you have to take a small hit to let customers realise what they value before it becomes indispensible to them. Henry Ford would have built a more comfortable horse carriage if he had asked the customer what to do. The distinction in marketing terms is between “True value” (product features) and perceived value ( How the customer sees it)
“There’s more than one way to ask the customer and more than one way to interpret the answer, if you listen with an open mind, sometimes you will be surprised pleasantly.”

4. We can’t have a discussion on customer experience without discussing the brand.
There are many definitions out there of a brand and I’ll leave that to those with little to do, for me the important point is that expectation which a customer carries as a result of the brand. That is what drives her through our door or to our site.
Let’s not gloss over the word “expectation”. Whether you are playing poker, editing movies, or doing magic tricks for your children, you will quickly realise that everyone, and that includes market researchers, sees what they expect to see, hears what they expect to hear and feels what they expect to feel. Most people could probably say yes to that statement glibly, but very few would appreciate the profound power of it.
In a previous blog I described the experiment when scientists used MRI brain scans to identify the increased satisfaction enjoyed by a coke drinker who had poured it from a branded can into a branded glass over that of another drinking it from a plain glass, all in stark contrast to the memorable testimonials of thousands who preferred Pepsi over coke when offered both in unmarked glasses and could only focus on taste.
Expectation is created in many ways, but primarily by the chatter of others and the perceived opinion of peers. That is the territory of Brand managers, Marketing people and Social Media experts.

The key Point here is that creating an expectation associated with your product is the most powerful way to create value for your customers and often the cheapest and mot certain way also.
“Innovation is critical, but don’t confine it to the engineers and inventors, the ultimate playing field is inside the customer’s head”

5. Customer Lifetime Value is not an old, or boring idea it has never been more relevant, or more critical.
One of the first things we tend to look at with a new product is a breakdown of the cost of product, cost of selling it and gross margin. The cost of selling a product usually surprises newcomers to the field.
In competitive markets with a lot of equal offerings a small price advantage can drive large sales increases so price is critical and it is driven primarily by cost. i.e you cant reduce price below a level that is profitable. In most markets price is sensitive and if it isn’t then investors are sensitive to margins, earnings and dividends. In all cases no business can indefinitely carry unnecessary cost and in a competitive market. Sooner or later the competition will do it and steal a march. Of course there are many pricing strategies and this is not a discussion on price
The money you spend on marketing and selling your product is critical to the success of your product, yet it comes under less scrutiny than any other budget apart from the CEOs expense account.
Let’s say you sell 1m units of a product at £100 retail. Your production cost is 20 and your marketing/selling costs are £30 operating costs are £40 and net profit is £10
That’s 100m t/o, 30m spent on selling 40m operating profit and 10m net profit

Suppose you convinced 1% of your customers to recommend the product to a friend
Now your t/o is 101m selling and operating costs stay the same and net profit is 11m.
That’s a ten percent increase in earnings- a darling of the markets if you can repeat it.

Let’s say that you have a Million customers, every customer has to be replaced after 4 years and they pay £1000 a year for your product. That’s t/o of £1b
To maintain that t/o you have find 250,000 new customers at a cost of £1000 each
That’s £250m a year in marketing/selling costs.
Now suppose you are so nice to these customers that they stay for 5 years instead of 4

Now your costs are reduced to £200m a saving of £50m
if your net profits were, for arguments sake, £100m on £1b now they would be increased to 150m a 50% increase in earnings. What would that do for your stock?

These are simplified figures used to demonstrate a point, so lets not get into a investment analysis discussion. The message is clear:
“Treating your customers well enough to retain them a little longer can deliver huge dividends while enlisting them onto your salesforce is the next killer app and make no mistake about it.”
That means paying attention to the user journey long after the “order to pay “ stream has completed.

Using information to support the entire customer journey

Previously

The customer journey  begins when she becomes aware of your existence and never ends, though it is at its most fruitful when she places an order and subsequent orders.

Previously we discussed the folly of looking at “Last Click” as the beginning of this purchase journey, the reality is that it began some time in the past when she stumbled on your business either through a friends, in a blog, or via a search or advertisement. In reality every purchase is generally precluded to a greater or lesser degree by a process of discovery, comparison, discussions, eavesdropping, information gathering, price comparison and leading finally to an order being placed.

Whether and when that order is placed will be contingent and whether she found sufficient information to support a decision, what information she found, what advice she got, what her peer group are doing whether she is in front of her favoured device for ordering, whether she has the cash available yet  and a probably many more issues. For example it matters little that she made her mind up on day one, if she wont have the cash until her salary clears in three weeks. It wont matter how good a deal you offer her if all her friends are advising against your product and so forth.

It is never possible to know all of these inputs and be aware of the state of play, but at least being aware of what it takes to sell an item is very important in determining what steps you take to improve that user journey in a way that is profitable. Below are some examples of information you may collect and use to improve the user experience and deliver revenue upside. This will of course vary from one situation to another.

  1. It begins with being found. You must know where the hungry crowd are going and make sure your food stand is right in their path. Being there when they are hungry is just as important as part of serving your customer as it is to your revenue targets. How to do this is a little off topic for today.
  2. Making sure that the gossip they hear and the advice they receive is unlikely to be negative is critically important. The means of promoting positive vibes in social media are well documented and to a lesser degree we know of business that can help deal with negative comment when it occurs.
  3.  Making the right first impression is critical. The expectation you set is a key metric against which your performance  will be measured.
  4. Becoming memorable and easy to find again is now a key goal. Any way of beginning a relationship that allows you to communicate further is great, getting the customer to download something that will act as a reminder for them is also very valuable. E.G. a useful app for their phone.
  5. Storing a cookie that helps you track their consequent visits and actions will make it much easer to judge their likely needs at any time.
  6. Running multivariate tests allows you to not just find out which inputs drive the most orders, which combinations pf inputs are most successful. This drives very accurate views of customer behavior and allows you to optimise everything.
  7. Once you understand the average customer journey you can provide content and services that help the customer at key junctures while updating your understanding of where they are at with their buying process.
  8. Understanding a little more about the type of product they shortlisted and what they rejected may also help you to understand their needs and preferences.
  9. Knowing the times of week, day, month, or year when they are most likely to make a purchase may help you in selecting an irresistible offer.
  10. Knowing which devices they use for purchase may help you to time your offer better

Here is a simplified example.

Background

My company sells widgets to consumers and the customers come form all walks of life. They purchase from the ecommerce channel. There is a lot of competition online  and customers tend to switch suppliers regularly as offers change. Price is important, but its not the whole picture.
We use advertising via keywords to drive customers to landing pages where they find information on exactly what they searched for. They can also follow links to the main site where they can  learn more

Mrs Jones

Our best customer is Mrs Jones. She uses search engines a lot but not just for finding products but also searching the news and gossip sites. She talks to a lot of people on forums and uses them extensively for advice before purchasing. Mrs Jones enjoys the purchasing process so she does not mind seeing plenty of offers, but she is rarely swayed from her initial choice. Often she decides what she wants and then goes looking for proof that she is right.

After she first selects a product, we know she is giving it strong consideration because she then visits our comparison charts and follows links to some of our competitors.

Our strategy

We think she trusts us because we are not hiding from our competitors and we give her honest comparison. We also help her out with the evidence she is looking for.

We have her email in an opted-in list and we know when to send her a little extra information if she goes quiet. We have a clickstream that identifies a quest (product she searched for) and the different types of investigation she did so far, so we can guess where she is in the purchasing journey.

Sometimes, when she goes quiet, it means she has bought elsewhere, but often she is just waiting to get paid or some other reason, so we keep in touch, but we are careful not to upset her. We rely on her to visit again and to recommend us. On average she makes five visit before purchase.

She is very influenced by social media so we spend a lot of effort on maintaining a good reputation.

Our content is tagged to match the different stages in the quest such as price comparison, features comparison, evidence gathering etc. These tags help us to develop the clickstream that places her on a purchase journey. Because she has purchased before, she is able to purchase with a single click.

Pre-visit

She visits an exhibition  where she sees our stand and meets a polite person who gives her a free pen.

She searches google for comments and finds a positive attitude towards us and our products

First visit

She spends some time reading the general information, downloads a calculator tool and leaves via our comparator to go to a competitor site.

 

Still collecting information

The following week our advertising network presents her a little reminder advert while she is on a competitor site and she returns to ours. This tells us she is still actively and seriously searching and we are high on her list

Decided now

She returns at the weekend and spends some time on the cost of ownership calculator using her tablet.

We know that she likes to purchase using her PC and she might still feel like this. We also believe that price is the only thing now influencing her.
We email her a very hot offer that needs a response before Tuesday and we give her a special hotline for telephone advice promising no switchboard and 12 hours a day of service.

Finally an order

She immediately calls our sales staff explaining a slight issue she has yet not resolved. The sales staff are able to put her mind at rest and she places a n order there and then. It is completed in seconds and she has an email confirmation

Delivery and service continues

Delivery occurs on Wednesday and our service staff phone her unexpectedly to talk her through getting started seeing that she expressed concerns. She expresses her delight with the service.

Recommendations

On  Thursday our sales staff call to make sure she is OK and ask her if she would be happy to recommend us on a social network, she agrees readily and goes public with her satisfaction. This has three important implications:
1. We are committed to keeping Mrs Jones happy.
2. Mrs Jones has publicly praised us and it would be extra hard for her to ever contradict this.  She will make allowances if ever called on to do so.
3. Others who see her comments will be encouraged to do business with us

We have not just sold a product, we have bought a supporter and gained valuable advertising of the best kind. If we worked out the Cost of Goods Sold on customers like Mrs Jones, it would be in low or even negative figures.

 

 

Are you wondering if a recommendation engine is the next big purchase for you?

Some retailers in particular have made a great deal of extra profit through offering clever recommendations to their customers so much so that the notion of the recommendation engine came of age in the last three or our years spurring a wave of new offerings form software vendors and plenty of noise in the blogosphere. Before you go running off to buy one or even nurturing plans t build one, you should give due consideration to exactly what you expect it to do for you and from there you will be better equipped to decide if it is the right thing and finally what type of engine you need and how to acquire or build the right thing.

Ins and outs of recommendations and personalization

Perhaps the oldest and best known recommendation engine is the one used by Amazon.com. This is sometimes claimed to be responsible for 35% of sales. If that’s the case then it’s not hard to see why there is a strong interest from the ecommerce community.  Every customer who selects s product then receives a number of recommendations to other products she may like . Since the customer s usually there to browse, she can live with the annoyance of being sold to and is statistically reasonably likely to find the suggested product worth looking at even if it were driven by a schoolboy randomizer function. A portion of that 35% would undoubtedly be equally achieved by a placebo tool and I would strongly recommend some experimentation before spending large sums.

The down side of recommendation engines can also be potentially substantial.  When Microsoft first experimented with personalization on their website they were a leader in innovation on the internet. I was a regular user at the time  and I remember being frustrated by my inability to find something that a colleague was recommending me to. The fact was that when I visited, my cookie told them I was of type A and these widgets were only of interest to type B. I t  took them about a year to realize their mistake and loosen off the personalization  rules.

In the past year I have had similar experiences with Google search. It is now so focused on commerce that it sends me results it believes I want to see rather than a list of cold hard facts that I want and need.  These are only the few occasions when I became aware of the filters. How much of my online activity is tailored  to a weird misconception of me created by a mad algorithm. Even I don’t have a great idea of what I’ll like tomorrow and that’s how I like it.
Do you want to risk excluding products from your customers because Mr customer looked at something last year that suggests he would not be interested in X. Imagine buying a Vegan book for your best friend and never again being offered a meat menu. Ugh!

 

Types of recommendation engine and what they can do

The Amazon, or Netflix type of engine with which we are all familiar is sometimes referred to as content based because it uses knowledge of your stock database (i.e. content) to decide what Ms H might like and make a recommendation.

The simple version is that:

Product (a) has been tagged to be about [1,2 and 3] Product (b) has been tagged to about [3,4 and 5] You looked at products A and B from a long list therefore there is a strong likelihood you will like other products about [3]  but possibly also about [1.2.4 and 5].
Of course the algorithms are somewhat more complex, but hopefully  you get the gist.

It can also mine the records of previous customers and genuinely say, people who selected product A also Bought product Y and Z. This will have a reasonable potential to be useful also to the customer.

Provided you are searching with intent, this type of implied logic can quickly build a useful picture of what to recommend. If you are just browsing then this interference could be just plain annoying.
The most reliable statistic however when it comes to shopping is that the more things a customer sees the more she is likely to spend, so even the mistakes are not that serious. Remember also that what works for books or movies may not work so well for other products or services.

The key to this type of engine is that it needs little knowledge of you the customer, it takes as inputs knowledge of the content and of what you searched for and how you reacted to the search results.
That is good behaviour, it has no preconceptions and it takes you at face value based on what you do.

Other engines receiving a lot of attention now are referred to as  collaborative filtering  engines.

These engines use vast amounts of data collected in various ways to form opinions about you and use those opinions to show what they think you will like. Some of the data in uses is controversial third party cookie data that is collected without your explicit permission.

Every action by a customer is a piece of information that potentially says something about that customer and the combination of these actions says a little more.

A simple method id to mine click streams and create segments based on identical click streams. Suppose that a high proportion of customers form segment B purchase product Y and your clickstream data puts you in segment B then guess which recommendation my engine will make.

Other information that may be collected and used against you is your interaction in social media. Who you are connected to says the type of people you like and a profile created from the commonest likes expressed by members of this group can be applied by default to you the moment you are seen to me a member of the group, any accuracy this profile has will then improve as a result of your on-going interactions with the engine via its recommendations plus any likes or other social sharing you, or your associates may express.

What is different about this method is that very little needs to be known about the content or stock in order to make predictions about the customers interests, it all comes from social an other interactions.

In theory at least, such an engine can recommend the white box to you with confidence, not knowing what it contains, simply because your colleagues whose tastes most resemble yours all bought the white box.

In reality a combination of the two methods works better because it uses some knowledge of the customer alongside some knowledge of the content to make a more intelligent match with a better likelihood of success. For the second method you do of course need to spend a considerable time collecting the useful data before you can make a start on creating recommendations, though many commentators grossly overestimate just how much data is required. Analysing 10m transactions (data points) wont necessarily give you a significantly better result than analysing 100,000 and certainly not sufficiently better to cover the extra cost.
A further problem with the big data approach is that data is time bound and therefore data form last year may or may not be valid this year. It may as easily be detracting form the result as adding to it.
A large amount of user data may well be relevant to a surprisingly small segment of heavy users whose needs are very different from most of the people you want to offer recommendations to.

People learn and change and the world changes. In 2014  attitudes are very different indeed to what they were on 2004 or even 2010. Most people I know have changed substantially in the past five years and a great deal of surfing is without doubt serendipitous. Google’s Z Moment of Truth is an interesting approach to discussing this subject.

Currently there are armies of start ups offering to find you the perfect restaurant, or movie or whatever and to my experience they are a long way from way form delivering on the promise even if I wanted to be told what to do.

My personal experimentation with Siri ended in abrupt divorce after just a few days and google’s sad attempts at knowing me have gone via the same route. The owner of  Ness claims as his mission to: “become that trusted source for people to find out the next thing they’ll like.”
Isn’t that what advertisers have been doing since the first  TV invaded the first living room?

There is no doubt that at a certain level for certain industries, recommendation engines can deliver substantial extra revenue and that is always a vote in favour, but the right one for the job is important.

For others a revenue upside sufficient to justify the cost can be achieved without causing damage to the user experience, but for some industries, some or maybe most recommendation engines will struggle to improve revenue an may well have serious detrimental effect of your user experience and therefore your brand

 

http://thebridger.co.uk/using-personalisation-cleverly-to-grow-your-customer-relationship-and-keep-your-sanity/

 

Requirements engineering strategy can make or break your project .

Part one – What would you like sir
Part two – Requirements,tests, training, help files
Part three – Why no project exists in isolation-what should be done
Part four – Business rules, Process rules, Process, Data, different viewpoints
Part five – Testing requirements is not optional
Part six  -Requirements strategy can make or break your project

If you’ve been reading my blogs on the subject of requirements, you will no doubt appreciate the level of importance I place on getting requirements right and then testing them before committing yourself to a contract or a technical specification.
Just in case you are thinking that this is in some way anti agile, then nothing could be further from the truth. Please read my blog on agile requirements engineering for a discussion on this topic.

Let’s take this opportunity to recap on why we do requirements. The simple fact is that once a development team start work on creating a technical specification, the till begins to ring up with costs and every time you make a change after this point more costs are added. The closer you are to rollout when you make the change, the greater the extra cost will be and that time phased increase is exponential, therefore the first purpose of requirements engineering is to contain costs and eliminate or reduce slippages.

The second purpose is to make sure that the end product delivers value by meeting the needs of the business precisely in order to meet or exceed the benefits targets set in the business case.

Getting requirements right therefore, is absolutely critical if you are to contain costs and deliver value

Developing a requirements engineering strategy

Each organisation and each project are different and requires a tailor made strategy for getting the requirements right by recognising, exploiting and working with the capabilities of the organisation.

To do this successfully, you will need a few things:

  1. A detailed analysis of the stakeholders involved in the as-is process in order to make interviewing efficient and thorough.
  2. A strong feel for the past experiences of the organisation in terms of successes or failures with software projects in terms of meeting business need and staying within budget and time constraints. This along with some tactful exploration of causes will give a strong steer about the current capabilities of the organisation to engage with a formal requirements process and to take seriously concepts like change control.
  3. A feel for the organisations ability to take on and successfully adapt change will help you decide whether and how far you might decide to upgrade their skills and attitudes to requirement engineering
  4. A good indicative plan that indicates the products to be created, their acceptance criteria, time scales for delivery and the amount of effort that will be needed from stakeholders.
  5. The understanding, agreement and total buy-in of key stakeholders to the proposed approach with full support in terms of making people and facilities available for the process

 

Stakeholder analysis for requirement engineering

Step one

Using RACI to identify stakeholders for the Requirements function.

In my blogs on business case techniques and models I discussed the importance involving the right stakeholders to gain buy-in and get a real picture of what success will look like.
In order to design the system, you will need a different type of stakeholder list, one that describes roles, responsibilities skills and communication. This list will give a clear view of where the knowledge skills and the responsibilities lie within the organisation and therefore point you at the right people to describe requirements and take responsibility for their quality. My favourite tool for achieving this is a well known HR instrument known as the RACI chart. Responsibilities Accountabilities, Consults, Informs and it provides a two dimensional view of a team that helps you quickly analyse the team for adequate skills, supervision, communication and quality control. It can be adjusted in form or emphasis to suit your precise needs, but the fundamental principal serves well in any circumstance straight out of the box.
Here’s an example;

RAACI example

The table above lists nominal roles across the top and Activities down the left, each cell is then marked with one or more of RACI to show who has what role in that activity.
The example is for a software development team, but if you were building a hospital it would list things like planner, builder, architect and activities like approve plans, complete design, etc.

A software project for a builders might involve, Construction director, surveyor, cost controller, quality controller etc and Liaise with clients, agree costs, sign off completion, etc.
Apart from being a huge benefit in helping to highlight the important stakeholders for your requirements process, the RACI chart also provides a quick but effective sanity check to make sure that your team is adequate and that there no duplications, conflicts or gaps.

Horizontal analysis of each task will quickly tell you whether there are sufficient doers and accountability exists and is in the right place.
Vertical analysis of roles will quickly highlight overworked, underworked, misplaced authority or responsibility and lack of or too much consultation and information.
Too much consultation stifles decision making and too little leads to dangerous decisions. Corporate culture can lead to busy bodies who don’t always follow through, or teflon in-trays. All of these issues need to be understood and ideally addressed ahead of system design, because the system will only perform as well as the people who use it.

Once you have created this chart, the next thing is to create a list of names with contact information to place in each of the role areas. If the project is large and/or complex, there may be more than one name in each box and there may be further sifting to do in order to resolve any doubts.
Don’t be surprised to find skeletons in the cupboard and lack of agreement over who does what. If you uncover these issues, now is the time to discuss them at a high level and attempt to get them resolved ahead of requirement gathering.

Step Two
The second step is identify the Actors, the people and systems that carry out tasks as part of this process and the sources of all the skills, knowledge and decisions required to complete the process successfully. This will generally begin with the R people in your RACI chart and will usually expand to people who do the work with or for the R person. E.G. the R may be in your chart as the Technical architects for design, but on investigation, you may find a DBA, a SysAdmin and many others who play key roles and these people also need to be interviewed.
The actual actors may be more usefully broken down to specific skill sets or disciplines as opposed to specific roles. E.G you may have Document writer, Interviewer, modeller all of which are actors within the BA function. This approach gives a slightly more abstract approach that lends itself to reorganising roles for efficiencies, or to adapt to a new system.Never be tempted to accept the assurance that a supervisor can tell you the whole story and you don’t need to talk to her/his charges. If anything this should be seen with an element of suspicion and tactful verification is a good idea.

To better understand and communicate how the various roles influence the outcome of a process, it is sometimes useful to create a fishbone chart adapted slightly for the purpose. Here is how i like to create it:

Stakeholder fishbone chart

The fishbone can be made as complex as you need it to be in order to show all the influences on your new project. The thing that becomes apparent very quickly is that whether they are aware of it or not, most of your stakeholders rely on others for skills, support, knowhow, data, administration and other inputs that can potentially prevent the processes progressing and hence you need to know about it and to resolve, or risk manage it.
Your fishbone chart will immediately tell you who to talk to and by way of a bonus, it will very quickly highlight the areas of influence, both positive and negative when it comes to rolling out.

Organisational culture and requirements process maturity.

Once your target interviewees are lined up, your next step is to hold a number of short interviews and/or an interactive workshop to discuss past projects, how they went and how the organisation reacted at the time and now. Asking them what they felt went well and what went wrong will go a long way towards gauging what will go down well, or will meet with resistance in terms of methodology.
The aim is not to deliver the perfect project, by your standards, but to deliver the best you can within the capabilities and expectation of the clients.

The last critical aspect of this phase is to get a good feel for their favoured mode of communication within this area. Do they have requirements documentation that they worked with in the past? Are they comfortable working with it? Does technical documentation create silos that exclude valuable stakeholders to the benefit of technical people .

This aspect is vital, because the quality of decisions they make and their perception of you as a trusted adviser will be directly proportionate to your skill in communicating the concepts clearly to them. Not only that, but your ability to get their attention at all let alone hold it long enough to achieve a useful level of consultation will depend on not confusing or alienating them with technical or complex instruments of communication or use of unfamiliar jargon.

A clearly communicated strategy and indicative plan

Once you have gathered your information it is time to make some preliminary decisions about how to take it forward. How to collect information, how to verify your information and how to communicate back to stakeholders ahead of defining the requirements.

Identifying these tasks, the stakeholder and support staff needed and the order and dependencies of the work, it is time to add this to a simple Gantt chart that defines high level activities, high level milestones and products to be delivered.

Understanding, agreement and total buy-in.

If you have done your homework and worked in a consultative manner you will be very confident before presenting the plan, that it will be understood and supported and will be accepted by your stakeholders.
You should present:

  1. An introductory explanation of the approach with reasoning
  2. A list of products you propose to produce backed by a simple concise explanation of how and why and samples.
  3. A Gantt describing the timeframes and milestones.
  4. A simple, high level risk assessment
  5. An indication of the time commitments required of key stakeholders.

At the end of this you should answer questions and ask for their full support backed by a sign-off of the plan.

Ed Taaffe is a senior Consultant in Business Improvement through technology and Hi-tech Product management.

Reblog this post [with Zemanta]

The health warning attached to agile is nothing more, it’s not a reason to ignore agile thinking as a powerful tool.

As a manager who entered software engineering as the agile movement was gathering pace and returned to management in the systems world, I find it amusing when other disciplines jump on the bandwagon as it were. I also find it encouraging, but I would have serious concern if it were my business and here is why:

  1. Agile is not what journalists say it is

In much the same way that we probably misinterpret and ingratiate many movements,  Agile is credited with a lot of stuff that was never  though t of,  never intended and for the most part never happens.
A bunch of software engineers (several in several places) to decide how they could “take back control” of the software process in business.  They knew the power thy held in their hands, but could see more and more of it ebbing into middle and senior management determined to make them “blue collar”.  At the same time service businesses like IBM and Microsoft were looking for ways to reduce the paperwork and improve communication in the software transaction. The two things were married and produced a clever, but unruly child that requires careful monitoring and constant praise.  Agile was never a strategy, but a bun fight and it is far from over.

2.Fiiddling with KPIs is not changing anything.
 In my early days as a marketer I was taught that if you are not meeting KPIs, it is easier to fiddle with the KPI than change your behaviour. This is a fairly universal principal of business and nowhere is it more fitting that the world of software.” I want it and I want it now, but I am not sure what it is and couldn’t communicate it to you even if I did . In any case, what if it doesn’t work?”  I’ll claim you didn’t deliver and /or you didn’t do it right. That is the scenario driving software engineers to do away with specifications and contracts and make the product owner responsible. Unfortunately, what they didn’t spot until too late was that in doing this, they sold their souls and became blue collar again, but that’s another story.  Now they estimate in Tee-shirt-sizes and look upwards and to the left as they declare how many tee shirts they completed this month. If they don’t like the design they simply say it can’t be done or will cost too much. That is roughly where the tyre meets the road toady

3. A racehorse designed by a vet and an art student will make a very ugly camel at best

I was a product Manager for a few years, it is a natural progression for a marketer cum software engineer and I know some of the best around the globe. Some are redundant and others struggling. Why? Who needs professional when you have the seat of your pants? I am not suggesting that Agile teams with a nominated Product Owner can’t or don’t produce good stuff, simply that; successful products need a market sized opportunity, or it matters not how good they are, products need to meet an unmet need and to do it so that they are a more attractive proposition that their competitors and so forth.
What I am getting at is this; iPhones are successful because they were driven forward by a brilliant visionary who had the courage to show us what we are missing and make us love iPhones. Bill Gates did this for the PC. In an agile world neither of these products would have happened.
The iPhone would have been an ugly mishmash of half-finished features rushed out to an imaginary deadline and reflecting the current tastes of the Lead developer and the Product Owner, usually in that order.  I suspect the PC would have been a typewriter with a glass screen and a coffee holder.

  1. Lying in the water and thrashing your arms about is not swimming.
    Anybody who seriously wants to make a change for the better needs to start not with the technology, but with goals, develop strategies and work their way down to the details of how.  If that happens to be agile, wonderful, there is then a powerful chance that you will make it work.
  2. Beware the evangelists, many are trainee terrorists

Understanding the shortfalls of anything is the first sign of expertise and even love. Don’t fall for the preachers they have alternative motives. Somethings and some teams lend themselves well to agile, but not everything and worst of all, when a good agile team has been doing the same thing for a year or so, they are no longer agile, they are simply where they were before with slightly different rules and rituals.

  1. Agile is a state of mind not a book or a methodology and must be championed and driven form the top down by people who know what they are doing and why they are doing it.

Stage gates are still imortant even in Agile product maanegement

When I was a developer in the late (90s, I had the benefit of having started my career in marketing and combined with an overdose of enthusiasm and self-confidence, I thought I knew better than anyone about how the product should be designed.

Not that many years later I was in charge of Product development and planning and with the benefit of some training and mentoring from some of the best, I was acutely aware of my frailty when it came to predicting the present, let alone the future for long enough to make a business case deliver and keep a whole building full of us in employment.

The fact of the matter is that because there is huge demand today does not mean there is means,  nor does it mean there will be demand, nor means next year. A market opportunity is a different can of works than a product opportunity and the first stage gate must at the very least ask the questions; Is there a strong possibility that this opportunity will deliver the benefit’s we need? Are we able to do it? Is it right for the brand or will it confuse our customers about who we are? what impact will it have on other products we have that may be rising stars or in the long tail generating strong revenue?

Of course you probably have a few more in mind, but those few alone are sufficient to point out that there is serious strategic thinking to be done before you scribble an elevator pitch on the wall of the developers den and go off to lunch.

As we progress through a software development project, we often come up with new problems we had not identified, or problems that are simply a whole lot bigger than expected. They may not always be technical, but could be regulatory changes  etc.

When these things happen and the likelihood of achieving an acceptable offering and still being able to return a profit is threatened, then there must be a stage gate process.

When the early launches are not achieving the type of results we anticipated and we don’t know why, there must surely be a strong argument for stepping back from it and taking an informed and sober view of the chances for success and the opportunity cost and potential reputation cost of a failure. Of course there is always an opportunity to handle these issues on the fly and do it fairly well, but there are times when a more sober and considered approach is a good idea.

 

 

 

The DNA of CHAOS. Why software projects, peace negotiations and football games all come up short on occasion and the true value proposition of agile

Why Business analysts, Product Owners, negotiators, political reformers, scientists and many others are doomed to repeat the same mistakes until they drown in in their failures.

  1. Most people in a work situation, and indeed in most situations, live in a consciousness that is deliberately falsified to reflect (a) what they believe is expected of them by peers, and (b) what the asker of a given question wants to hear. This is a conditioned response and nobody’s fault.  Terms like “Group Think” attempt to explain it, but none really explain the phenomenon well and the long version is just beyond our scope in this piece. Market researchers are trained to try and avoid it, Politicians and advertisers learn to harness it. If you have the role of finding out what  a business wants to do so you can get the software designed to support, it, you will be faced with the same  aggravated problem. Asking and listening simply won’t get the job done.
  2. For reasons I don’t claim to understand fully, most people have a favourite solution in their minds for most problems they are aware of. Expensive  housing – they may say; “Stop immigrants coming in”.  The call centres is too expensive and eating our profits- they may say; “ Close the call centre and let them use Twitter”. Always there is an obvious political, or financial gain for the responder from this solution. If it is a senior executive, or politician, there will be cleverly argued points and tables of figures that make her pet solution look like a no-brainer.
  3. Michael D Cohen  coined the phrase “Garbage Can Theory” to describe this phenomenon which permeates politics, business, the playground and just about every area of human life crippling every endeavour to improve our lot century after century. If you are trying to get your proposal through programme and investment boards, or trying to agree a solution closer the coal-face you will drown unless you understand this phenomenon and take steps to work with it.
  4. Many researchers and writers have agreed strongly that the human brain makes virtually all decisions in a matter of time between milliseconds and 1 to 3 seconds, but then spend enormous amounts of time and effort researching in order to prove their decision was correct. Most people are entirely unaware that they do so.  This of course is not news to Scientific researchers who often publish the theory before beginning on the research.  In the academic world where we have to start somewhere this can be a useful approach. Product managers and entrepreneurs also start with the theory and are difficult to budge even when wrong, yet these leaders are necessary to get something started at all.
  5. People are conditioned to follow a strong leader and will convince themselves of their support for any theory put forward by this leader. This is largely an extension of  point one, but worth being aware of all the same when venturing into any group to arrive at a shared view.

These Five forces combined thwart just about every effort to form a good business strategy with technology in mind, agree a solution and still be in agreement when the solution is completed to specification even a few months later, let alone after a year or more. Waterfall and other methods of managing the software delivery process all rely on a simplistic view that if you gather enough requirements from users and stakeholders, you can then haggle over priority and build the killer solution to solve everyone’s problems. In reality, there is rarely any agreement, there is even less concern over business outcomes and near- total focus on the many garbage can solutions. Nothing this project could deliver is ever likely to meet the satisfaction of more than a small proportion of stakeholders, the others are always going to be loud in their dissatisfaction and only by sheer luck is their ever going to be serious business benefit.  This project, however well the techies perform will be added to the list of “failures” and the cause will be placed at the CIO’s door. Failure to focus on goals Developers and technical architects are driven by developing exciting new toys. That is their package in the garbage can. Techies ultimately want and are entitled to, clear definitions of what they must build.  They then build and test precisely what you asked for and give it to you working  very well. Few people would try to argue with that, let alone succeed.  The problem is that what you asked for does not always address the problem you needed solved and in fairness the developer or indeed the CIO is not to blame. There is one technique that all accomplished negotiators agree on whether brokering peace in a war zone or negotiating a business contract. YOU MUST KEEP THEM FOCUSED ON GOALS NOT SOLUTIONS! If you don’t, the garbage can will take over.
One cause  of the problem is not using the effective solutions we already have,
The need to focus on goals rather than solutions is not  new to the software industry by the way.  MSP, Prince2 and TOGAF are frameworks I am personally very familiar with, all of which, when used correctly, focus on outcomes required, the solutions that will best deliver those outcomes and the details of each solution, in that order. BABOK and the ISEB framework for business analysis also follow these same principals and are very clear about it. The problem is that few people make any attempt to use their training effectively, the training is rushed and focused only on answering multiple choice questions to gain a certificate and of course the majority of people involved in the business have no training at all. This same scenario, by the way, applies equally to Scrum. Scrum solves some of the problem Scrum attempts to alleviate the problem in a different way and with a similar level of success. Scrum is invented and owned by the technical community as a direct response to these issues plus the difficulty of communicating the complexity of technology with business people. The Scrum team wants only to be given an unambiguous requirement so they can do what they studied for, build great software. They place the onus on the business to appoint, or work with a Product Owner who will be seconded to their team and take all responsibility for the definition of the Product/System.   Communication issues are solved by very short Sprints of just two weeks typically and using prototypes and working software to describe things to the product owner in place of documents. Initially this arrangement solves the problems for the Scrum team, there is no ambiguity about what they must build, they do as much work as the Product Owner wants and stop when he is satisfied. There is no room for dissatisfaction with the IT team. Likewise for the Product Owner, if he is unsure, he can build something , test it on real world scenarios and then make adjustments until it delivers the right outcomes or scrap it altogether.
Inspect and adapt solves problems with lack of knowledge, but also masks others Scrum tries to tackle the strategic questions by replacing strategic decisions and documents with Inspect and adapt. For online products and for certain internal systems, this works extremely well because you can build the minimum, get it working and in use, learn from success or failure and add a little more, remove some, or start again. The risks are lower, failures are smaller and less expensive and the problems with groupthink, daydreaming and garbage cans are alleviated by an early sobering splash of cold reality that comes with release of a Minimal Viable Product. What is often wrong with this approach and fatally dangerous, is the lack of a strategic layer to the thinking process. Inspect and adapt is a close cousin of the OODA loop as defined by Colonel John Boyd. The principal is very simple that in a dynamic environment, he who has the most clear view of the situation always wins given a reasonable ability to make decisions.  Pilots flying inferior jets were seen to consistently win dogfights  because they had a better view from the cockpit and could make better decisions faster.  Developing OODA  (Observe, Orientate, Decide, Act ) and teaching it to pilots made significant improvements over giving them a detailed instruction, or a faster Jet because as the military have always recognised, in the field, the operative has superior knowledge in the heat of battle and must have the right level of autonomy and confidence to decide and act. No military  has ever cut loose a person, or team, or unmanned drone to inspect and adapt and just stay out there. There has to be a clear goal, a Definition of Done, Principals that guide it in the field and a strategy for completing the mission. Then Inspect and Adapt comes into play and in the same way that the human brain itself learns,  the actions  in the field can feed all the way back to sometimes cause a change in strategy or even a revision in Principals, but  actors in the field DO NOT  GO AWOL. Principals and strategy come first and Inspect and Adapt is a response to exceptions , it is not the  plan.

The real reason why Agile/Scrum is winning friends in the world of software and gaining attention beyond these confines

I say Agile/Scrum because when most people speak of agile they are really referring  to Scrum. However Scrum has encompassed much from other Agile implementations and definitely does not hold the sole franchise on Agile methods. The first problem Scrum had to address was the great chasm between Business leaders and Technology leaders.  Various versions of the same solution see the business seconding someone to the Scrum team, or colocation of business and technical people working on the same project. This is a perennial problem sometimes referred to as silos and secondment or colocation of cross discipline teams is a useful technique that helps, but doesn’t alone solve the problems. In the case of Scrum it forces the business into making decisions for better or for worse and living or dying by them. That helps techies to get on with building stuff, but it doesn’t, of itself, help the business much with getting the strategy right. I.E they produce more stuff, but don’t necessarily solve more problems. The next problem Scrum had to address is helping the business in dealing with the unknown. The speed of change is much greater in the past few decades driven by the fast pace of disruptive technology. Organisations are faced with problems they can’t analyse easily and therefore can’t solve. They also find that when they know the problem, they don’t have an obvious solution. This is the situation Scrum was designed for in its entirety. A Scrum team can make a best guess and very quickly get a potential solution in front of customers, Inspect and Adapt and return quickly with an improved effort until the right solution is reached by trial and error and then it can be built upon until it reaches its optimum scale. The biggest problem addressed by Scrum and that one that is causing all the excitement is the almost  total  lack of leadership throughout modern businesses (the past 10 years). Few would lament the passing of the great Command and Control organisations, but with them went whatever semblance of leadership we had in organisations. In the Flat organisation leadership in name is often cited, but leadership in deed is sometimes frowned upon or even punished. In the 80s and 90s business leaders bullishly wrote books about leadership and motivation. These were charismatic people with an instinct for what made others tick.   Kenneth Blanchard, Peter  Drucker, Stephen R Covey and their like gave us the ideas and the passion to change things and build things. We somehow lost this passion and insight in the transition.  Scrum is a beacon in the dark, because it values people and their interactions over trails of evidence. It values achievement over promises and it values self-improvement  and job satisfaction. Here and there Scrum teams are springing up that capture attention by their high productivity and ingenuity and win the envy of their colleagues for their enthusiasm and Joie de vivre.  People who see a good Scrum team in action want to be part of it or learn how to make that happen in their business.  That is the real reason why the buzz of excitement continues and why Scrum experts are being invited into other areas of business.