How much data is “Big Data”

For many, this question is almost an irrelevance. The question that should always start the conversation is “ What do you want to achieve?”, yet in my personal experience it never has and when I have introduced it I have been made to feel uncomfortable. Many feel that they must have a big data project in their portfolio and the why? and how? is of less importance. A high proportion want answers to fairly simple questions they can’t currently get answered and are lead to believe that answering those questions is indeed big data.
Let me make this very simple. With few exceptions, there is only one reason why you might want a Big Data solution: Because you have so much data that anything less could not analyse it and provide solutions in the timescales you need.  There are two key elements here; Timescales and volume of data to be analysed.

Timescales is the simplest one so let’s deal with that first. There tends to be two timescales: 1. Instant response and 2. Non-urgent responses. The latter is by far the more common and is typified by the “Data warehouse” approach. The former is typified by the “search engine” scenario.
Although the search engine appears to be providing instant response, in reality it is merely searching well-ordered indexes that have been populated at a leisurely pace, so in fact it does not differ as much from the data warehouse scenario as one might at first think.
The data warehouse is a model of efficiency where the questions are carefully defined in advanced, most of the processing done and the answers stored away until needed.  Often further processing is then carried out at the point of consumption.
Again you may be thinking that there are more parallels than differences between the two approaches apart from all the hype. You’d be right.

What does Big data do that is different?

Well the term as we understand it, owes its existence to Google’s own solutions to the search engine problem. Perhaps another penny has dropped for you now.  Hadoop, Map-reduce and all those sexy terms refer to a simple and very powerful approach to getting a huge job done efficiently.

The infrastructure relies on the idea of dividing each job into smaller jobs and continuing to do so until each is quite manageable and then delegating them to different machines. If you’re a software engineer, think Jackson.  A simplified view might be that you have five people doing operational work and a manager coordinating that work and responding with a single answer to his sponsor. If you ever attended management courses you will surely remember this type of organisation. Well that’s the big idea.
Why is this better? Well it allows a vast, unlimited number of servers to work on bits of the problem at the same time,  thus speeding up the time to complete. This allows one to demand immediate answers to questions that are more efficiently dealt with over a longer time-frame and there lies the risk.
Is Big data Machine Learning?
No, it definitely is not, but of course it can be useful for doing this. However, it is very important to understand that there is a plethora of tools, many free and some you already have in the toolkit such as excel,  that can very effectively carry out machine learning tasks if you take a little time to learn them. Not only  is it  infinitely easier to learn and carry out such analysis on tools like SQL server, Excel, etc. than it is to spin up a big data factory on AWS and become a data scientist just to find out if it will rain tomorrow.

Very few questions you are likely to want answers to require anything more than traditional statistical approaches or even simpler BI reporting that can be carried out very effectively on stunning data volumes and extremely complex problem domains with tools like EXCEL (try SOLVER or explore the many regression functions), POWERBI, KNIME, RAPIDMINER, MATLAB, OCTAVE, Google FUSION TABLES, TABLEAU. Many are free and very good tutorials can be found online. The best thing about these tools is that you can test your hypothesis and decide whether a major project is worthwhile.

How big is big?

Well there are truly big problems and yours may well be one of them, but the vast majority of questions can be answered with a well specified windows server or your personal preference.
rememeber alos that remarkably small samples are known to provide extraordinaty insihts that improve very little when expanded.
For a better technical analysis than I could offer have a look at this very good blog. It gets to the point much faster than I do

As usual, comments only via email. No new subscribers being accepted at this time.

Image result for bot wikipedia

I wont be long-winded about this, I’ll discuss it via email with anyone who is interested, but I’ll break with my usual mode and come straight to the point.
A great many people who know little at all of machine learning  and even less about people and many more who are simply  oblivious to the potential consequences of their words are talking about the miraculous things we can expect from Machine learning.

What is ML in a nutshell?
Academics break ML into two modes:  Supervised and Unsupervised.
In the case of the former we give the machine a large corpora of content and ask it to decide what will happen next, or to find other similar instances. A translation service for example  begins this way and learns after a while to translate without help.
In the latter case, we give it a body of content and ask what it thinks of that.. Google search is an example of this approach and it simply makes sense of what it finds.

Often we give it a few hints like “Classify this for me and establish links” as in Google search. This would be a “Classification problem”. We might on the other hand ask it to read the racing papers and decide who will win the four o’clock today. This would be a “Regression problem” because we are asking it to look at the past and predict the future. Yes all of this is highly condensed as promised, if you are an expert you don’t need my explanations.
Understanding what the customer will want next year, predicting the weather, finding Oil under the sea, predicting tumours, the challenges are endless and the rewards enormous.

What is the loop of self-destruction?
The loop happens when, thanks to social media, a good, but no a sole example, the machine begins to make judgements that influence the data and then discover exactly what it predicted.

As with humans this will give it the machine equivalent of a big head and possibly some citations and will lead to even greater confidence and fewer checks and before anybody spots it, it I all too late.
If any Movie producers out there are stuck for an idea, I am available to help with the plot. Here is a simple example we are all aware of:
Joe Gel, and Josephine Lotion our dear friends, represents an enormous body of intelligent and informed people who spend most of their waking hours  checking back with their phones for reassurance. Joe searches Google for Tom Raspberry, his favourite politician and receives a huge list of pages. The ML in google notes his interests and begins sending him dozens of articles about Tom Raspberry, what he says and does and what people say about him. Unwittingly Our pal Joe has become astonished by the fact the whole world seems obsessed with Tom R and realises subconsciously how important to is be aware of Tom R. He begins to tweet and have the odd Facebook conversation about something he read. Immediately the ML in Facebook and the one in Twitter hone in his apparent obsession with Tom R and all begin to bombard him with content and introduce him to thousands of people with the same problem. Poor Joe.

Now our Machine does a Recce to see what are people talking about and it discovers that millions are talking and reading about Tom raspberry and concludes that tis is the way to keep the customer happy so it ups its game and heightens the emphasis. It also confidently announces that Tom R will undoubtedly be unstoppable in the forthcoming election.

Joe and Josephine realise the importance of not standing in the way of a social crowd and are not about t be shunned and subconsciously they begin to take more interest in the positive stories about Tom which now triggers the Machine to filter their feeds and search results and friend recommendations etc more toward the positive . You don’t need me to finish the plot. There is only one way this is going. Imagine if the secret services relied on this kind of information to brief their bosses. But they do, don’t they.

You may well think, as I do , that despite the  shear “wrongness” of rigging democracy, whether by design or accident, it matters little who is elected anyhow. In that case imagine the same scenario when the machine turns its hand to guiding change in a government department or a large business , or guiding product development or even finding the cure for cancer. If you would like to see many better examples with a strong scientific analysis, check out Weapons of Math destruction.

One wonderfully simple yet highly destructive outcome of ML that I have seen up close is the  call centre  automated system that recognises your telephone number, calculates your value as a customer and decides if you will be answered, how long you will have to wait and whether you get to speak to somebody skillful.  Just to update my card details for a £20 a month hosting service, I had 11 hours of my time wasted, had my service disrupted and was threatened by a bot with £150 fine to put the service back on.
I hate to disappoint you, but if you have ever had an IM conversation with a patient lady on the support portal “That was no lady” nor was it my wife, that was a distant cousin of Cortana.
If she did not know the answer, or more likely the question, you were never going to be served.
If you are wondering what might happen to your pension, your job and your home if these guys get involved in stock trading, well take a look here  According to a 2014 report, sixty to seventy per cent of price changes are driven not by new information from the real world but by “self-generated activities”.

It’s not all negative by any means. I actually do use ML to predict the winners of tomorrows racing with a consistent level of profit. When I get it wrong, usually after a late night of programming with insufficient testing, my winnings disappear very quickly into someone else’s pocket and I sit up and take notice.
I sincerely hope that someone starts sitting up and taking notice soon  of the impact of poorly programmed Bots that are already beginning to increase risk for the most powerful nations on earth.

Why you need to pay attention to customer experience

Why you need to pay attention to customer experience

Next        What to watch out for when researching and testing journeys


The chicken and egg question always fascinated me. When it comes to business models I find the same conundrum with customers and profits. Michael Porter once said that the purpose of a business is “to create value for customers”. Although we all assume it was inferred in there, he never bothered to mention profits.
The reality every business faces however it that creating value comes first and monetization follows.

1. Compare the debacle of the great Thatcherite privatisations to the often maligned success story of dot com.
In the UK we have a raft of privatised utilities who still have not “got it”, they still think in terms of Oligopoly, force, bullying, price rigging. They think and act like tax collectors. The total innovation from all of them over two decades could be written on the back of a credit card along with a full list of their happy loyal customers.
Amazon, ebay, Paypal, Google and many more have on the other hand built world beating businesses on the back of profitless customer satisfaction and only now are monetising these business models. They operate at P/Es up to 500 and have no shortage of investors.

The message is clear, the customer is king and until you can demonstrate value to them you don’t have a business model.
“ Sooner or later regardless how much cash you have stashed away, you will learn to create value for customers or fail.” We even see this law apply itself to dictatorships.

2. What is customer value and how can you create it?
The biggest possible blunder any business can make is to quantify customer value in terms of product features. I cringe when I see these neat spreadsheets listing product x competitior1, competitor2 etc and how well they score on each (in the marketing trainees opinion).
Customers buy an experience, even hard nosed corporate customers. That begins with the interaction with “People” in the supplier side, or “friendly” and human like ecommerce site and carries right through to anticipating delivery, opening the package, using it for the first time, bragging to friends, interacting with support and many more. Many of these are remarkably powerful influencers and even though supported at times by product features, most of the time they are a separate source of value, or indeed antagonism.
Next we return to the chicken and the egg.

3. Does customer experience exist without customer value and who foots the bill?
The problem here is thus: If you ask the customer how much extra they would pay for their phone to float up out of the box on a mechanism with a Jingle playing, be fully charged, sense the old phone and offer to copy the contacts and messages etc in a sweet voice, accept a voice answer. The customer might well offer a price that made this simply not feasible. However, when that same customer experiences it once, the likelihood is that she won’t want to be without it and when she hears her friends talking glowingly about it, it becomes a must-have at almost any price. Soon it is talked about and develops a cult status and then we have a brand value to take into account. That’s a whole new ball game.
I’m not suggesting we deliver high quality customer experience at all costs, I’m simply saying that you must understand the true value and what people do is far more revealing than what they say.

The point I’m making here is that sometimes you have to take a small hit to let customers realise what they value before it becomes indispensible to them. Henry Ford would have built a more comfortable horse carriage if he had asked the customer what to do. The distinction in marketing terms is between “True value” (product features) and perceived value ( How the customer sees it)
“There’s more than one way to ask the customer and more than one way to interpret the answer, if you listen with an open mind, sometimes you will be surprised pleasantly.”

4. We can’t have a discussion on customer experience without discussing the brand.
There are many definitions out there of a brand and I’ll leave that to those with little to do, for me the important point is that expectation which a customer carries as a result of the brand. That is what drives her through our door or to our site.
Let’s not gloss over the word “expectation”. Whether you are playing poker, editing movies, or doing magic tricks for your children, you will quickly realise that everyone, and that includes market researchers, sees what they expect to see, hears what they expect to hear and feels what they expect to feel. Most people could probably say yes to that statement glibly, but very few would appreciate the profound power of it.
In a previous blog I described the experiment when scientists used MRI brain scans to identify the increased satisfaction enjoyed by a coke drinker who had poured it from a branded can into a branded glass over that of another drinking it from a plain glass, all in stark contrast to the memorable testimonials of thousands who preferred Pepsi over coke when offered both in unmarked glasses and could only focus on taste.
Expectation is created in many ways, but primarily by the chatter of others and the perceived opinion of peers. That is the territory of Brand managers, Marketing people and Social Media experts.

The key Point here is that creating an expectation associated with your product is the most powerful way to create value for your customers and often the cheapest and mot certain way also.
“Innovation is critical, but don’t confine it to the engineers and inventors, the ultimate playing field is inside the customer’s head”

5. Customer Lifetime Value is not an old, or boring idea it has never been more relevant, or more critical.
One of the first things we tend to look at with a new product is a breakdown of the cost of product, cost of selling it and gross margin. The cost of selling a product usually surprises newcomers to the field.
In competitive markets with a lot of equal offerings a small price advantage can drive large sales increases so price is critical and it is driven primarily by cost. i.e you cant reduce price below a level that is profitable. In most markets price is sensitive and if it isn’t then investors are sensitive to margins, earnings and dividends. In all cases no business can indefinitely carry unnecessary cost and in a competitive market. Sooner or later the competition will do it and steal a march. Of course there are many pricing strategies and this is not a discussion on price
The money you spend on marketing and selling your product is critical to the success of your product, yet it comes under less scrutiny than any other budget apart from the CEOs expense account.
Let’s say you sell 1m units of a product at £100 retail. Your production cost is 20 and your marketing/selling costs are £30 operating costs are £40 and net profit is £10
That’s 100m t/o, 30m spent on selling 40m operating profit and 10m net profit

Suppose you convinced 1% of your customers to recommend the product to a friend
Now your t/o is 101m selling and operating costs stay the same and net profit is 11m.
That’s a ten percent increase in earnings- a darling of the markets if you can repeat it.

Let’s say that you have a Million customers, every customer has to be replaced after 4 years and they pay £1000 a year for your product. That’s t/o of £1b
To maintain that t/o you have find 250,000 new customers at a cost of £1000 each
That’s £250m a year in marketing/selling costs.
Now suppose you are so nice to these customers that they stay for 5 years instead of 4

Now your costs are reduced to £200m a saving of £50m
if your net profits were, for arguments sake, £100m on £1b now they would be increased to 150m a 50% increase in earnings. What would that do for your stock?

These are simplified figures used to demonstrate a point, so lets not get into a investment analysis discussion. The message is clear:
“Treating your customers well enough to retain them a little longer can deliver huge dividends while enlisting them onto your salesforce is the next killer app and make no mistake about it.”
That means paying attention to the user journey long after the “order to pay “ stream has completed.

The trap of settling for mediocrity

The trap of settling for mediocrity

Without boring you with a treatise on macro-economics, I can only get my point across by saying that, not only is every professional and tradesperson in our world challenged to know less and possess fewer skills, but most business we are employed by will have not a single individual who understands what the business does from end to end even at a fairly high level. I have met more than a few CEOs who didn’t consider it important to them to understand the business they were charged with, but focused on a small handful of KPIs to get him out of jail this year.
The guy who scans a label and uses another tool to locate the good, before handing it to someone else to pack and pass on to the loader . . .  is no less dumbed down than the Financial controller who receives the chart identifies the KPI and barks out an instruction via email.
Mediocrity is everywhere and it is tempting to succumb to it but if you are not seduced by warnings of the long term damage to our world then pay some heed to this:

Skill and competency is 98.5% self-esteem and 1 % knowhow, the other stuff isn’t even worth talking about. Although I have never met you, I know without any fear of contradiction which of these you most fall short on. The reason I know is that a confident person can be taught almost anything while others can make almost anything go wrong.   Have you ever noticed how football players go on runs of scoring and always return from a lean patch to push the boundaries even further.  The tool they are using is deep seated self-belief.  Sure, some aspects of this are probably learned from Mom and Dad or even inherited in their genetic make-up, but don’t start making excuses again. The world is full of superstars who had no running start and plenty who got their inner strength from facing adversity.

Average is a dangerous illusion

All my life I have sought relaxation, release and inspiration from games of chance. I know the numbers and I totally understand what you have to do over the long haul to come out ahead. I am happy to say that I have been coming out on top since my early twenties and if there is one thing you must learn if you are to stand a chance against bookmakers, poker players, or any other adversary it is this.  The averages are the things that make you profits, but to achieve an average that keeps you ahead you have to perform above average by a good margin at all times. You don’t just need a margin to win by, you also need a margin for error.  Then it all works out like the good book says. Sadly it is true that the level we aim at is always a shade higher than the one we actually achieve

Average, for all it’s flaws, is above and beyond the capabilities of so many and only because of their mind set. The Peter Principle : Why Things Always Go Wrong Paperback – August 19, 1998 was and still is  a master class in systems thinking.
Peter observes a phenomenon we are all familiar with, i.e. that once people begin on the promotion ladder, they will then, as members of the hierarchy, be promoted until they reach a level where they are totally incompetent.    Naturally this comes as no surprise to the reader, what still seems to surprise many is that the outcome of this apparently innocuous behaviour dictates that everybody in charge in too many businesses, then and indeed today is incompetent to some degree. Had I made this statement before presenting the evidence, you would have dismissed me as a crank and convinced yourself not to listen, or found another escape, but now that you have read this far, surely the conclusion is inescapably true even if the reasons for such behaviour are overly forgiving in many texts.

The average, therefore, is not only something that describes systemic incompetency, but over the long haul, it is indeed the result of settings ones sights somewhat above that level. Are you still content to get up tomorrow and set out to be average?
At the end of every good lesson is a stiff test and here is yours: What did you take from this little blog?

  1. The world of business is messed up and there is no point in even trying, what can I do?.
  2. The world of business is full of incompetent people, the opportunity for competent positive people has never been greater.

You can mark yourself on this one

Confirmation bias at the speed of light

The extraordinary world of the trader really opened my eyes

Image result for halo effect

Most readers will already be aware of confirmation bias and will no doubt believe they have it under control. Good for you. If however you are trader, or gambler the likelihood is that you have got it down to a fine art and it has become a heuristic behaviour.  What I mean by this simple, is that you’ve been doing for so long and become so attached to the theory that you are smarter than the rest, that now your brain does it in auto-mode without your intervention or even awareness.

Recently I spent some time developing trading tools for people earn a  living trading movements on a betting exchange.  The betting exchange, for those who are not familiar is a simplified clone of a stock exchange and is very similar to currency trading .

Some of these people act as bookmakers accepting bets form the public, while others simply trade the movements in the market or “SCALP” the market.
I was amazed to hear bookmakers of some standing use the phrase, “ Let’s get this one beaten”.
Quietly testing the intent of this over a period of time I realised that they actually associated their attempts to attract money and lay the runner with the runner losing.  Naturally they would never admit to it openly and they all know that such a thing is impossible, yet they daily select a weak runner and go about betting against it with the hope that it will be beaten and regularly it loses and bit by bit the brain has begun to associate tis intent and action with the outcome. Given they will be laying an 8: shot that really as a 1 in 12 chance of winning, the see it lose 11 times out of 12.
What is happening here is the brains own “Inspect and adapt learning process”, taking what it sees at face value and jumping to dangerous conclusions.
For years renowned physicians dispensed useless or even dangerous treatments to unsuspecting patients while convincing themselves and the patients that it was having a positive effect. After all, some of these people recovered. I wonder how much of this still goes on?
Economist and ex trader Max Keiser recently did a TV show on economics where he talked about stock traders believing they were changing the economy by their actions when in fact they are simply gambling with liquidity against other traders.
The UK government has been conspicuous in their inactivity in terms of fixing the economy, or even banking since the crash. They have sat back and conserved the status quo for all intents and purposes, yet George Osborne makes speeches in which he associates his office with a “reported “ improvement in the economy. Has it improved? If so, has it improved beyond a possible “ gently rising tide”?  Does he actually believe it?  What do you think?

Given the power of such self-delusion, would it be a shock if we found traders fixing things like exchange rates, or would it be far-fetched to imagine it might then stretch to bribing politicians and officials.
If those same people had gone into the munitions business instead of banking, do you think they might find a way to start wars? Do PMs and  generals ever admit that they achieved nothing ( best case scenario) or created a catastrophe?

Systems thinking

Negative bias the damage it can cause

Using information to support the entire customer journey


The customer journey  begins when she becomes aware of your existence and never ends, though it is at its most fruitful when she places an order and subsequent orders.

Previously we discussed the folly of looking at “Last Click” as the beginning of this purchase journey, the reality is that it began some time in the past when she stumbled on your business either through a friends, in a blog, or via a search or advertisement. In reality every purchase is generally precluded to a greater or lesser degree by a process of discovery, comparison, discussions, eavesdropping, information gathering, price comparison and leading finally to an order being placed.

Whether and when that order is placed will be contingent and whether she found sufficient information to support a decision, what information she found, what advice she got, what her peer group are doing whether she is in front of her favoured device for ordering, whether she has the cash available yet  and a probably many more issues. For example it matters little that she made her mind up on day one, if she wont have the cash until her salary clears in three weeks. It wont matter how good a deal you offer her if all her friends are advising against your product and so forth.

It is never possible to know all of these inputs and be aware of the state of play, but at least being aware of what it takes to sell an item is very important in determining what steps you take to improve that user journey in a way that is profitable. Below are some examples of information you may collect and use to improve the user experience and deliver revenue upside. This will of course vary from one situation to another.

  1. It begins with being found. You must know where the hungry crowd are going and make sure your food stand is right in their path. Being there when they are hungry is just as important as part of serving your customer as it is to your revenue targets. How to do this is a little off topic for today.
  2. Making sure that the gossip they hear and the advice they receive is unlikely to be negative is critically important. The means of promoting positive vibes in social media are well documented and to a lesser degree we know of business that can help deal with negative comment when it occurs.
  3.  Making the right first impression is critical. The expectation you set is a key metric against which your performance  will be measured.
  4. Becoming memorable and easy to find again is now a key goal. Any way of beginning a relationship that allows you to communicate further is great, getting the customer to download something that will act as a reminder for them is also very valuable. E.G. a useful app for their phone.
  5. Storing a cookie that helps you track their consequent visits and actions will make it much easer to judge their likely needs at any time.
  6. Running multivariate tests allows you to not just find out which inputs drive the most orders, which combinations pf inputs are most successful. This drives very accurate views of customer behavior and allows you to optimise everything.
  7. Once you understand the average customer journey you can provide content and services that help the customer at key junctures while updating your understanding of where they are at with their buying process.
  8. Understanding a little more about the type of product they shortlisted and what they rejected may also help you to understand their needs and preferences.
  9. Knowing the times of week, day, month, or year when they are most likely to make a purchase may help you in selecting an irresistible offer.
  10. Knowing which devices they use for purchase may help you to time your offer better

Here is a simplified example.


My company sells widgets to consumers and the customers come form all walks of life. They purchase from the ecommerce channel. There is a lot of competition online  and customers tend to switch suppliers regularly as offers change. Price is important, but its not the whole picture.
We use advertising via keywords to drive customers to landing pages where they find information on exactly what they searched for. They can also follow links to the main site where they can  learn more

Mrs Jones

Our best customer is Mrs Jones. She uses search engines a lot but not just for finding products but also searching the news and gossip sites. She talks to a lot of people on forums and uses them extensively for advice before purchasing. Mrs Jones enjoys the purchasing process so she does not mind seeing plenty of offers, but she is rarely swayed from her initial choice. Often she decides what she wants and then goes looking for proof that she is right.

After she first selects a product, we know she is giving it strong consideration because she then visits our comparison charts and follows links to some of our competitors.

Our strategy

We think she trusts us because we are not hiding from our competitors and we give her honest comparison. We also help her out with the evidence she is looking for.

We have her email in an opted-in list and we know when to send her a little extra information if she goes quiet. We have a clickstream that identifies a quest (product she searched for) and the different types of investigation she did so far, so we can guess where she is in the purchasing journey.

Sometimes, when she goes quiet, it means she has bought elsewhere, but often she is just waiting to get paid or some other reason, so we keep in touch, but we are careful not to upset her. We rely on her to visit again and to recommend us. On average she makes five visit before purchase.

She is very influenced by social media so we spend a lot of effort on maintaining a good reputation.

Our content is tagged to match the different stages in the quest such as price comparison, features comparison, evidence gathering etc. These tags help us to develop the clickstream that places her on a purchase journey. Because she has purchased before, she is able to purchase with a single click.


She visits an exhibition  where she sees our stand and meets a polite person who gives her a free pen.

She searches google for comments and finds a positive attitude towards us and our products

First visit

She spends some time reading the general information, downloads a calculator tool and leaves via our comparator to go to a competitor site.


Still collecting information

The following week our advertising network presents her a little reminder advert while she is on a competitor site and she returns to ours. This tells us she is still actively and seriously searching and we are high on her list

Decided now

She returns at the weekend and spends some time on the cost of ownership calculator using her tablet.

We know that she likes to purchase using her PC and she might still feel like this. We also believe that price is the only thing now influencing her.
We email her a very hot offer that needs a response before Tuesday and we give her a special hotline for telephone advice promising no switchboard and 12 hours a day of service.

Finally an order

She immediately calls our sales staff explaining a slight issue she has yet not resolved. The sales staff are able to put her mind at rest and she places a n order there and then. It is completed in seconds and she has an email confirmation

Delivery and service continues

Delivery occurs on Wednesday and our service staff phone her unexpectedly to talk her through getting started seeing that she expressed concerns. She expresses her delight with the service.


On  Thursday our sales staff call to make sure she is OK and ask her if she would be happy to recommend us on a social network, she agrees readily and goes public with her satisfaction. This has three important implications:
1. We are committed to keeping Mrs Jones happy.
2. Mrs Jones has publicly praised us and it would be extra hard for her to ever contradict this.  She will make allowances if ever called on to do so.
3. Others who see her comments will be encouraged to do business with us

We have not just sold a product, we have bought a supporter and gained valuable advertising of the best kind. If we worked out the Cost of Goods Sold on customers like Mrs Jones, it would be in low or even negative figures.



The evils of groupthink and the risk to all Social media users

AKA “ Say hello to someone”
Power to think
The most spectacularly creative people in many fields are often introverted, according to studies by the psychologists Mihaly Csikszentmihalyi and Gregory Feist. They’re extroverted enough to exchange and advance ideas, but see themselves as independent and individualistic. They’re not joiners by nature.” NYtimes SUSAN CAIN JAN. 13, 2012

Power to innovate
No great life-changing, or world-changing invention has ever come about as a result of market research. Whatever the agile evangelists tell you, the reason for success in this area is not what they think it is. Asking the customer is important but it is not innovation. It is a stark and unavoidable fact that when Henry Ford asked a carriage owner what he needed from transport, he replied along the lines of; ”A horse that doesn’t foul the road”, or “A nice warm fire in my cab”. He never mentioned anything resembling a motor car and he never would either.
Try to imagine our world without the legacy of Henry Ford
In fact I would argue that lack of this type of thinking is why we have moved from the single blade razor to two, three, four blades and who knows where this will end. No there is definitely nothing much to be gained by innovators from hanging around in large groups or spending their day exchanging messages of one sort or another or asking customers. Watching customers is different, but that’s a different blog.

Decision making power
Making decisions is the toughest part of running a business, a department, a team. It is tough because if there was a safe answer it would not need a decision. Let’s face it if you are in a room on a cold winter’s day and the door blows open allowing rain and cold wind into the room, nobody needs to make a decision about what to do, now do they?
Decisions happen when there are several options and imperfect information and someone needs to make a call. Decisions involve risk management and playing the averages, but most of all they require taking responsibility and having the confidence to do so, i.e. leadership.
So what happens when you ask a committee to make a decision? Well the commonest thing is that they decide to close the door, just after the last piece of furniture has blown away and the rain has shorted out the electricity, but often they make no decision, or if they all belong to a particular religion, group, union just about anything could happen, it could end in a brawl. If there is a bully in the group they will guess what he wants and pretend that is what they want, but he won’t of course be accountable. The simple answer is don’t do it if there is any other option. Fake it if you absolutely must.
“In big business and national politics, groupthink has emerged as one of the most damaging spectres of our time. Unquestioning corporate cultures have been blamed for scandals ranging from Volkswagen’s emissions-test rigging to the sexism controversies that have surrounded Uber and American Apparel. Last year, commentators accused hive minds on social networks of scuppering open debate in the run-up to the UK’s general election. But does groupthink always require such a large canvas to flourish?” Economia 22 June 2016

Power to compete
Is there any industry left where it is not an accepted policy (among the group) to recruit from within the industry and close competitors? . If you believe Volkswagen were the only ones fiddling the figures you are even more naïve than I could have credited you for. Check out where the senior executives of all the other Auto businesses learned their trade and developed their MO and you will have a map of who else is doing it. The fact is that behaviour patterns, but especially abuses spread like wildfire across industries and then gradually leak outside. Why? Because to continue with bad behaviour, it has to become accepted as the norm. You don’t need me to make a list, everyone has a huge list of such behaviours they could rattle off, if you are stuck, just start with a google search for “*ism”.
Competing in any market in the modern uber-competitive environment demands that you differentiate strongly and then communicate this difference effectively. Naturally it has to be a difference your customers value, but we have limited space to fill today . . .
Imagine how successful you will be at differentiation if all of your thought processes are shared (perhaps hourly) with your competitors and you can’t remember the last original thought you had?

Power to recruit
A note from the head of business change to HR reads:
“ I need a consultant to help with this important programme of change. Find me someone from within the industry, we need someone who will fit in with the culture and . . .”
The only organisation I ever worked for that is no longer around used to do exactly this. Their Achilles heal was that they were all from an industry where they had never learned to make decisions, but whenever they sought an outsider to help them, it began with “You guessed it “.
Social media could well be the most destructive force in modern times
Let’s forget about politicians playing to the baser forms of humanity just to get votes and all the evils we are forewarned about. Think Brexit, Trump , I’ll leave the list to you.
When is the last time you saw a real conversation happening on any social media platform? Well I can answer it for you with a fair degree of confidence. The answer is either never, or not for a number of years. Let’s ignore all the Hootsuite jockeys and other fakes dumping garbage on to the internet all day every day and those paying a King’s ransom to look important, the remainder are following the “groupthink” approach to social media in their sleep. Pretending to be interested in the things they are pretending to be interested in so that they appear to fit the imagined norm and
saying the things they guess the other “normal” people would expect them to say, or rather believe they are expected to say. I hope I didn’t confuse you. You could struggle along with these thoughts for a few minutes and probably end up dizzy, or maybe you’d prefer to grow a few brain cells and “say hello to someone”. Here is the big secret waiting to get out; when they are not faking it, they are all human just like you, they are just scared to let it show. That is my new theme for the remainder of 2016, “ Say hello to someone”. Or just in case you haven’t been paying attention: communicate

Two important rules of the learning organisation that you won’t study in an MBA yet they are ignored at great cost.

The learning cycles described by    Sekar Sethuraman CISSP,CISA,CISM,CGEIT,CIA,PMP is in my view an excellent starting point that would most likely be a significant achievement to formalise and encode into almost any organisation.

My personal experiences apart from academic interest are around :

  1. What we often call “lessons learned” i.e. the constant adjustment to circumstances and to the perceived results of our previous actions.
  2. The impact of swarm intelligence on our ability to learn and teach.

Lessons learned, maybe we should not be doing it at all?

In the area of Project Management, few organisations do anything at all about “Lessons learned”  though virtually all would express a regret about this. In truth , doing nothing formally is not doing nothing after all and hence informal learning continues. Is that better, or worse? Well it’s not a clear-cut answer.

In the past year I returned for a period to the area of managing risk in an uncertain and volatile environment with vague rules and little explicit information.  A classic example of this environment is day trading , or any kind of investment banking activity, bookmaking, military activity, espionage, football  and a long list of less glamorous situations.  Football is too tricky for this discussion  because  there is almost no conscious decision making involved.
The reason I choose to discuss these more obviously volatile environments is because they are more like real life only sped up enough to trigger human emotions and  for people to  learn from trends  and responses  that otherwise  might remain hidden. i.e. because you see the results of your actions soon enough to make an association you have a chance to reflect on the actions and the outcomes that in normal, snail’s pace life , would remain hidden from most people .negative false

Actions and results are two key elements of all learning. John  Boyds OODA loop is a wonderful example of this.  What Boyd recognised is the need for “ Sense making” and this is the key, because learning without doing this effectively is to learn things that are patently wrong. The equivalent Is to put arsenic in your coffee cup.
In a fast moving environment we see every day people who pushed  a button a and felt a shock up their leg. Like the pigeon learning to select the right beans, he stops pushing the button, because he assumes a relationship between the two. How quickly he makes this assumption and how rapidly he reacts is directly related to his self confidence and very quickly you can see the cookie crumble to a pile of dust.  Burned out traders are almost as common as arrogant and broke ex-traders.
The answer lies in the ability to ignore what you hoped or expected to see, question what you do see and only act on proven information while filing the rest away for another day. The ability to do this is much scarcer than you might think.

Lessons learned can be formally handled in a project management environment and these lessons ingrained in culture at which point the will become pervasive until they need to be superseded.

That leads us neatly to the other area:

Swarm Intelligence or Hive Mind has most of us  firmly in her grasp.

Swarm Intelligence , or Hive Mind as I prefer, or in simpler terms culture is a far more pervasive and more potentially damaging force than most observers realise , in particular when it comes to learning.  Ask Paddy Power Bookmaers.- Paddy Power Left ‘Red Faced’ After Early Payout on Greek Vote. They trusted the wisdom of crowds and learned an expensive lesson.

Surowiecki had a bestseller and started a wave of books that appeared to discover something new in old wisdom , only to be widely discredited later.

According to Jesse St Charles of University of Tennessee at Chattanoogai, there are specific rules that define a swarm or flock:
1. The rule of separation. Think of a flock of birds flying in close formation but never  make contact physically. There is an unwritten rule that keeps them a certain distance apart and that rule alone defines where they go. Watch the starlings over Rome about this time of year.

  1. Cohesion. The birds all use the same patterns of flight and movement and even squeak and defecate in unison.
  2. Alignment. They gauge their direction by where everyone is going and align themselves
    4. Type recognition. A flock of starlings will never allow a crow to join, nor will he try

Once these rules are in place, the bird has waived all control over his own brain and simply follows the pack.  In all group based creatures this can be seen and it mostly likely stems for the safety of being in a group and ideally close to the centre.

Parallel this to the Stanford Prison experiment when a group of well bred and highly intelligent students form the top 5% or so of Americans were given roles and group structure in two opposing groups; Prisoners and warders and left to enact this for the benefit of a study.  If you don’t know what happened, I urge you o read this:

I am not commenting about the scourge of starlings in European cities, or the capabilities of the human given the right opportunity, I am simply pointing out that once an individual has identified his or herself as belonging to a particular group a large part o his/her brain is surrendered to the perceived group intelligence and the power of the written or unwritten rules of that group prevent learning anything that is contradicts the group in any small way


If you are engaged in deepening the learning of your organisation, or team bear in mind two extra rules:

  1. Unless you adjust the culture of your hive so that learning and changing is a source of social acceptance and security, all you efforts will come to nothing.
  2. Sense making, for adults in a business means something different than in teaching and learning. People’s emotions play a huge role in how they perceive the effects of their actions, what they learn from what they see and even what they see. If you want to create a learning organisation, you must teach and assist people to collect and observe the results of their actions in an objective way, make and execute good decisions at the right time and police their emotions against rash reactions, or unearned self-doubt.


Perhaps the short description of all this is leadership, the thing mankind craves for more than anything


The health warning attached to agile is nothing more, it’s not a reason to ignore agile thinking as a powerful tool.

As a manager who entered software engineering as the agile movement was gathering pace and returned to management in the systems world, I find it amusing when other disciplines jump on the bandwagon as it were. I also find it encouraging, but I would have serious concern if it were my business and here is why:

  1. Agile is not what journalists say it is

In much the same way that we probably misinterpret and ingratiate many movements,  Agile is credited with a lot of stuff that was never  though t of,  never intended and for the most part never happens.
A bunch of software engineers (several in several places) to decide how they could “take back control” of the software process in business.  They knew the power thy held in their hands, but could see more and more of it ebbing into middle and senior management determined to make them “blue collar”.  At the same time service businesses like IBM and Microsoft were looking for ways to reduce the paperwork and improve communication in the software transaction. The two things were married and produced a clever, but unruly child that requires careful monitoring and constant praise.  Agile was never a strategy, but a bun fight and it is far from over.

2.Fiiddling with KPIs is not changing anything.
 In my early days as a marketer I was taught that if you are not meeting KPIs, it is easier to fiddle with the KPI than change your behaviour. This is a fairly universal principal of business and nowhere is it more fitting that the world of software.” I want it and I want it now, but I am not sure what it is and couldn’t communicate it to you even if I did . In any case, what if it doesn’t work?”  I’ll claim you didn’t deliver and /or you didn’t do it right. That is the scenario driving software engineers to do away with specifications and contracts and make the product owner responsible. Unfortunately, what they didn’t spot until too late was that in doing this, they sold their souls and became blue collar again, but that’s another story.  Now they estimate in Tee-shirt-sizes and look upwards and to the left as they declare how many tee shirts they completed this month. If they don’t like the design they simply say it can’t be done or will cost too much. That is roughly where the tyre meets the road toady

3. A racehorse designed by a vet and an art student will make a very ugly camel at best

I was a product Manager for a few years, it is a natural progression for a marketer cum software engineer and I know some of the best around the globe. Some are redundant and others struggling. Why? Who needs professional when you have the seat of your pants? I am not suggesting that Agile teams with a nominated Product Owner can’t or don’t produce good stuff, simply that; successful products need a market sized opportunity, or it matters not how good they are, products need to meet an unmet need and to do it so that they are a more attractive proposition that their competitors and so forth.
What I am getting at is this; iPhones are successful because they were driven forward by a brilliant visionary who had the courage to show us what we are missing and make us love iPhones. Bill Gates did this for the PC. In an agile world neither of these products would have happened.
The iPhone would have been an ugly mishmash of half-finished features rushed out to an imaginary deadline and reflecting the current tastes of the Lead developer and the Product Owner, usually in that order.  I suspect the PC would have been a typewriter with a glass screen and a coffee holder.

  1. Lying in the water and thrashing your arms about is not swimming.
    Anybody who seriously wants to make a change for the better needs to start not with the technology, but with goals, develop strategies and work their way down to the details of how.  If that happens to be agile, wonderful, there is then a powerful chance that you will make it work.
  2. Beware the evangelists, many are trainee terrorists

Understanding the shortfalls of anything is the first sign of expertise and even love. Don’t fall for the preachers they have alternative motives. Somethings and some teams lend themselves well to agile, but not everything and worst of all, when a good agile team has been doing the same thing for a year or so, they are no longer agile, they are simply where they were before with slightly different rules and rituals.

  1. Agile is a state of mind not a book or a methodology and must be championed and driven form the top down by people who know what they are doing and why they are doing it.

How much damage can negative BIAS do to performance and quality of life?

Whether a manager, a sports person, an investor, or a dustman It is vital to understand our personal bias and manage it .

Every decision we make requires some level of objectivity to stand a good chance of serving us well.
“You’re faced with around 11 million pieces of information at any given moment, according to Timothy Wilson, professor of psychology at the University of Virginia and author of the book Strangers to Ourselves: Discovering the Adaptive Unconscious. The brain can only process about 40 of those bits of information and so it creates shortcuts and uses past knowledge to make assumptions”
 A large part of each decision will utilise principals and strategies that we have developed over time. Without these assets we could not function, our thinking would be just too slow
It is nevertheless also the case that alongside of these positive assets we  develop bias that is not helpful or positive.
e.g. an Employer with race or age bias will miss recruitment opportunities resulting in lost production and opportunities. Its probably impossible to be human and not collect some of this negative bias, but it is possible and absolutely crucial to know what your bias is and be vigilant. Thats a different thing from beating yourself up over it. You’re human, you’re allowed to be imperfect, but if you deliberately fail to do what you know is the right thing, that is another situation altogether.
– A footballer who always gives up at the last millisecond against bigger players and limps off apparently injured with a “losers limp”.
– A poker player who gets over ambitious every time he wins and then loses big time.
– A recruiter who dumps CVs with 10pt or smaller type and misses some of the best talent.
– A golfer who panics at the top of the back swing grips and swings down with all his force running out of energy, line and balance long before it gets to the ball and will often continue to do this for his entire lifetime.
A belief that bigger people will be stronger, or more aggressive is a common misconception.
Believing in the god of luck instead of the mathematics affects all kinds of people, even statisticians can fall to it.
Being too lazy to adjust the type size before printing is unbelievably common, as is struggling with weak eyesight without seeking help.
Focusing on the map instead of the territory or vice versa is universal.
These are very easy problems to cure, but some can be a little tougher.
Here’s the simple map: 
1. Recognise and admit that you do it and the cost to you, or spot opportunities to leverage it in your favour.
2. Work out how to confront your negative bias head on.
3. Recognise any influences that work to maintain the bias.
4. Keep on until the habit is replaced with the right behaviour.
5, If you can’t get rid of it entirely then simply keep checking for it and adjusting.
6. Play to your strengths and expose yourself to the situations you are best at.
How to watch for and recognize personal bias.
Look at the things that went wrong recently, track down the bad decisions you made that lead to them. Be brutally honest and replay the decision process to see where it went wrong. There, lurking in the shadows, you will often find a negative bias driving the bad decisions.
Another useful tactic is to ask close friends or colleagues that you trust. Be prepared to be shocked.
Finally look for the classic forms of bias:
Confirmation bias: Only looking where you can be confident of finding the thing you want to find. Or seeing what you expect to see in spite of the evidence.
Anchoring: My only tool is a hammer therefore every problem looks like a nail.
Halo effect: This works in football so its bound to work in the kitchen, or I am good at A so I m bound to be good at B.
Overconfidence: I am so good, I cant be wrong, don’t question me, or advise me. If I accept help, I will look weak.
Groupthink:  It’s important to fit in with the others, especially the cool ones.
Outdated:    Clinging to principals that once were sound, but are now damaging. (That includes situations when you are right)
How to confront personal bias head on.
It may not be easy, because the driver behind a bias will often be an emotion such as  irrational fear, hatred, greed, etc. Fear is by far the most common of these emotions. The other very strong candidate is peer pressure i.e.  you have this bias because all the members of your peer group have it and it is part of the acceptance criteria. Sometimes it is conditioned from an early age.
How to identify the influences and counter them
The footballer in our example could join a boxing club and realise that he can easily hold his own against much bigger guys, but smaller ones can be very good. That would be a way to get rid of the irrational fear and open the door for a more confident player that went the last yard with full commitment.
A sound strategy that counters many forms of bias is this; when you have time to do it, seek a conflicting view to yours and study it. Either it will strengthen your conviction or it will open your eyes. In either case you will reap big benefits. An easy way to maximise this affect is working with your team to arrive at joint decisions and acting as chair. This externalsies the process for you and develops your team at the same team.
Ask yourself how friends and colleagues in your circles will react to your changed behaviour. Often you will recognise the source of your bias as a peer group who reinforce the bias. This can be a tough decision, but sometimes one of them has to go.
Often one of these bias’ will seem to be part of who you are and you are reluctant to change, but that is simply untrue. Bias changes over time all on its own without your intervention except not always in a positive and helpful way

How to keep at it until you win
Don’t expect miracles, you will slip a few times, but focus your thoughts on the positive outcomes you are expecting to achieve. The moment a negative thought about it begins in your mind, immediately replace it with something pleasant, or even downright naughty, but zap that negativity. When you make a mistake, pretend you have just had a triumph, celebrate and reward yourself so you stay positive. If you can find an admirable role model that will help an awful lot.
Once the benefits begin to accrue it will become a no-brainer.
How to be vigilant
Lets say I’m an investment manager and I handle billions in investors money. I have somebody data mining my trades to point out to me when a bias appears to be forming. He rushes to my desk and he says: “You have sold a day too soon on 70% of hour trades since last Monday, is there something you need to talk about?”
I think it over for a while, we look through the figures and I see that I had a heavy loss a week ago when I held on too long. I had been warned about the potential effects of a certain political situation and blamed the error on being too optimistic. I have been running scared  since and selling in panic.
Poker payers call this “On tilt”, For most of us, it is unfortunately part of our untutored makeup, but it is deadly in the world of a dealer and won’t do the rest of us any good either.
Once I know it exists and I can see the cause, fixing this is easy. I just watch the charts, follow the rules, feel the fear and deal with it.
The real damage that negative bias does is rarely mentioned
To fully understand how important this  subject is, you heed to delve a little into how the brain works.
Learning is a process of:
1. Observing our environment and absorbing communications and stimuli,
2. Making sense of the information we find is a process of comparing new information to stuff we already have stored away and passing it through filters to find out how we will classify it. These filters are made up of strategies and principals (positive bias) and negative bias. These are applied in the “fast thinking” subconscious part of the process. Consciously considering it all and applying logic.
3. Deciding how to classify the new information and whether we want to keep it?
4. Acting on your new knowledge means using it in the real world and this  is the final step in learning.  The problem here is that unless you are capable of honestly and objectively examining the feedback, your learning process is already broken and you are destined to swamp your brain with destructive conflicting rubbish.
How you classify information will govern whether or not you are able to reuse it. If you are carrying around filters that are wrong then you will spend your time rejecting valuable information and misclassifying critical knowledge. The damage continues and escalates.
Finally, when you act upon your new information and look for feedback, you will accept stuff according to your broken filters and the other broken bias such as conformation bias, group think and their mischevious friends.

Managing our personal bias is as important as brushing our teeth, but the rewards are much greater. It’s not about not making mistakes, it’s about watching out for them and taking action.
A video made by Google is revealing.
Word or concept associations in dictionary form is a key part of how humans store and retrieve information.
The speed at which we are able to memorise these associations  is far superior to any other form of information storage and retrieval.
Speak the name of a country and most of us can instantly reply with its capital city. It comes naturally.
When we make an association between grey hair and age or age and frailty, or obesity and laziness etc etc, we are then sowing the seeds of bias that is neither useful nor  helpful.
Take the controversial IAT test to help you discover your bias.