How much data is “Big Data”

For many, this question is almost an irrelevance. The question that should always start the conversation is “ What do you want to achieve?”, yet in my personal experience it never has and when I have introduced it I have been made to feel uncomfortable. Many feel that they must have a big data project in their portfolio and the why? and how? is of less importance. A high proportion want answers to fairly simple questions they can’t currently get answered and are lead to believe that answering those questions is indeed big data.
Let me make this very simple. With few exceptions, there is only one reason why you might want a Big Data solution: Because you have so much data that anything less could not analyse it and provide solutions in the timescales you need.  There are two key elements here; Timescales and volume of data to be analysed.

Timescales is the simplest one so let’s deal with that first. There tends to be two timescales: 1. Instant response and 2. Non-urgent responses. The latter is by far the more common and is typified by the “Data warehouse” approach. The former is typified by the “search engine” scenario.
Although the search engine appears to be providing instant response, in reality it is merely searching well-ordered indexes that have been populated at a leisurely pace, so in fact it does not differ as much from the data warehouse scenario as one might at first think.
The data warehouse is a model of efficiency where the questions are carefully defined in advanced, most of the processing done and the answers stored away until needed.  Often further processing is then carried out at the point of consumption.
Again you may be thinking that there are more parallels than differences between the two approaches apart from all the hype. You’d be right.

What does Big data do that is different?

Well the term as we understand it, owes its existence to Google’s own solutions to the search engine problem. Perhaps another penny has dropped for you now.  Hadoop, Map-reduce and all those sexy terms refer to a simple and very powerful approach to getting a huge job done efficiently.

The infrastructure relies on the idea of dividing each job into smaller jobs and continuing to do so until each is quite manageable and then delegating them to different machines. If you’re a software engineer, think Jackson.  A simplified view might be that you have five people doing operational work and a manager coordinating that work and responding with a single answer to his sponsor. If you ever attended management courses you will surely remember this type of organisation. Well that’s the big idea.
Why is this better? Well it allows a vast, unlimited number of servers to work on bits of the problem at the same time,  thus speeding up the time to complete. This allows one to demand immediate answers to questions that are more efficiently dealt with over a longer time-frame and there lies the risk.
Is Big data Machine Learning?
No, it definitely is not, but of course it can be useful for doing this. However, it is very important to understand that there is a plethora of tools, many free and some you already have in the toolkit such as excel,  that can very effectively carry out machine learning tasks if you take a little time to learn them. Not only  is it  infinitely easier to learn and carry out such analysis on tools like SQL server, Excel, etc. than it is to spin up a big data factory on AWS and become a data scientist just to find out if it will rain tomorrow.

Very few questions you are likely to want answers to require anything more than traditional statistical approaches or even simpler BI reporting that can be carried out very effectively on stunning data volumes and extremely complex problem domains with tools like EXCEL (try SOLVER or explore the many regression functions), POWERBI, KNIME, RAPIDMINER, MATLAB, OCTAVE, Google FUSION TABLES, TABLEAU. Many are free and very good tutorials can be found online. The best thing about these tools is that you can test your hypothesis and decide whether a major project is worthwhile.

How big is big?

Well there are truly big problems and yours may well be one of them, but the vast majority of questions can be answered with a well specified windows server or your personal preference.
rememeber alos that remarkably small samples are known to provide extraordinaty insihts that improve very little when expanded.
For a better technical analysis than I could offer have a look at this very good blog. It gets to the point much faster than I do

As usual, comments only via email. No new subscribers being accepted at this time.

Image result for bot wikipedia

I wont be long-winded about this, I’ll discuss it via email with anyone who is interested, but I’ll break with my usual mode and come straight to the point.
A great many people who know little at all of machine learning  and even less about people and many more who are simply  oblivious to the potential consequences of their words are talking about the miraculous things we can expect from Machine learning.

What is ML in a nutshell?
Academics break ML into two modes:  Supervised and Unsupervised.
In the case of the former we give the machine a large corpora of content and ask it to decide what will happen next, or to find other similar instances. A translation service for example  begins this way and learns after a while to translate without help.
In the latter case, we give it a body of content and ask what it thinks of that.. Google search is an example of this approach and it simply makes sense of what it finds.

Often we give it a few hints like “Classify this for me and establish links” as in Google search. This would be a “Classification problem”. We might on the other hand ask it to read the racing papers and decide who will win the four o’clock today. This would be a “Regression problem” because we are asking it to look at the past and predict the future. Yes all of this is highly condensed as promised, if you are an expert you don’t need my explanations.
Understanding what the customer will want next year, predicting the weather, finding Oil under the sea, predicting tumours, the challenges are endless and the rewards enormous.

What is the loop of self-destruction?
The loop happens when, thanks to social media, a good, but no a sole example, the machine begins to make judgements that influence the data and then discover exactly what it predicted.

As with humans this will give it the machine equivalent of a big head and possibly some citations and will lead to even greater confidence and fewer checks and before anybody spots it, it I all too late.
If any Movie producers out there are stuck for an idea, I am available to help with the plot. Here is a simple example we are all aware of:
Joe Gel, and Josephine Lotion our dear friends, represents an enormous body of intelligent and informed people who spend most of their waking hours  checking back with their phones for reassurance. Joe searches Google for Tom Raspberry, his favourite politician and receives a huge list of pages. The ML in google notes his interests and begins sending him dozens of articles about Tom Raspberry, what he says and does and what people say about him. Unwittingly Our pal Joe has become astonished by the fact the whole world seems obsessed with Tom R and realises subconsciously how important to is be aware of Tom R. He begins to tweet and have the odd Facebook conversation about something he read. Immediately the ML in Facebook and the one in Twitter hone in his apparent obsession with Tom R and all begin to bombard him with content and introduce him to thousands of people with the same problem. Poor Joe.

Now our Machine does a Recce to see what are people talking about and it discovers that millions are talking and reading about Tom raspberry and concludes that tis is the way to keep the customer happy so it ups its game and heightens the emphasis. It also confidently announces that Tom R will undoubtedly be unstoppable in the forthcoming election.

Joe and Josephine realise the importance of not standing in the way of a social crowd and are not about t be shunned and subconsciously they begin to take more interest in the positive stories about Tom which now triggers the Machine to filter their feeds and search results and friend recommendations etc more toward the positive . You don’t need me to finish the plot. There is only one way this is going. Imagine if the secret services relied on this kind of information to brief their bosses. But they do, don’t they.

You may well think, as I do , that despite the  shear “wrongness” of rigging democracy, whether by design or accident, it matters little who is elected anyhow. In that case imagine the same scenario when the machine turns its hand to guiding change in a government department or a large business , or guiding product development or even finding the cure for cancer. If you would like to see many better examples with a strong scientific analysis, check out Weapons of Math destruction.

One wonderfully simple yet highly destructive outcome of ML that I have seen up close is the  call centre  automated system that recognises your telephone number, calculates your value as a customer and decides if you will be answered, how long you will have to wait and whether you get to speak to somebody skillful.  Just to update my card details for a £20 a month hosting service, I had 11 hours of my time wasted, had my service disrupted and was threatened by a bot with £150 fine to put the service back on.
I hate to disappoint you, but if you have ever had an IM conversation with a patient lady on the support portal “That was no lady” nor was it my wife, that was a distant cousin of Cortana.
If she did not know the answer, or more likely the question, you were never going to be served.
If you are wondering what might happen to your pension, your job and your home if these guys get involved in stock trading, well take a look here  According to a 2014 report, sixty to seventy per cent of price changes are driven not by new information from the real world but by “self-generated activities”.

It’s not all negative by any means. I actually do use ML to predict the winners of tomorrows racing with a consistent level of profit. When I get it wrong, usually after a late night of programming with insufficient testing, my winnings disappear very quickly into someone else’s pocket and I sit up and take notice.
I sincerely hope that someone starts sitting up and taking notice soon  of the impact of poorly programmed Bots that are already beginning to increase risk for the most powerful nations on earth.

Amateur Excel wizards-the simplicity, scale and incredible consequences of their errors

Nobody imagines they can copy a bit of code off the internet and write the next messenger app, or heart monitor, but frequently people, whose position and professional standing suggests they ought to know more, make a hasty edit to an excel sheet in between meetings with extraordinary consequences.

* An accountant omitted a single minus sign resulting in their dividend estimate spreadsheet being out by a $2.6 billion.

* A simple cut and paste error resulted in TransAlta over paying $24 million for US power transmission hedge funds. “Cut and paste” can you believe it?

* In a written statement, Jayne Shontell, Fannie Mae’s vice president for investor relations put a $1.3 billion reporting error down to an “honest mistake in a spreadsheet”, but did not go into detail about the error.

* “ JP Morgan, was running huge bets (tens of billions of dollars, what we might think of a golly gee gosh that’s a lot of money) in London. The way they were checking what they were doing was playing around in Excel and not even in the “Masters of the Universe” style that we might hope, all integrated, automated and self-checking, but by cutting and pasting from one spreadsheet to another. And yes, they got one of the equations wrong as a result of which the bank lost several billion dollars”. Again, “Cut and paste” the mind boggles!

* Reinhart, Rogoff… in 2010 published an influential report, “Growth in a Time of Debt”, used by economists globally, but specially in US and Europe to argue against fiscal stimulus. Now anyone with a grasp of economics had to be wondering about this straight away, however, it was not until a student, Herndon spotted not just flaws but what looked like careless weighting of data ranges and blew the whistle that the scale and arrogance of the mistakes came to light. Who knows what impact these errors had on millions of human lives.

* A report published in January stated that poor spreadsheet protocols were primarily to blame for JP Morgan’s estimated $5.8bn (£3.8bn) of trading losses racked up last year from credit default swaps, including by a trader nicknamed “The London Whale”.

Spurned by a continuing list of catastrophes, regulators around the world, including the Basel banking committee and Britain’s Financial Services Authority, have focused their attention on spreadsheets as a key component of best-practice corporate governance.

You may not be running J P Morgan, but the figures you are dealing with have equally big consequences for you, your staff, your customers, your shareholders. The stark reminder here is that code is not for children, cut and paste is something we don’t even discuss in polite company, methodical approaches with sufficient testing and oversight is the bare minimum and I’m not even discussing regulation or compliance. Serious decisions should be made using reliable information that has been quality checked and is understood for what it is.
The average executive playing with a few formulas and macros in Excel could soon be risking a Jail sentence and certainly eventual ruin is inevitable.
Software engineering is now a mature profession and when allowed, let alone encouraged to do it right, the profession will follow proven procedures and well-trodden paths to ensure a very low risk of highly expensive errors. It is easy to understand the temptation for a subject matter expert to ignore the need for a software expert and just paste in some data, but increasingly this kind of foolishness is being identified and outlawed and it can only be a matter of time until “Honest mistakes” come under more serious scrutiny.
A few simple steps now and an achievable plan could pay big dividends in a short time frame and besides,  a seasoned pro can achieve more safely in a few days or even hours sometimes and not only cut out the risk and delays, but save you money into the bargain.

If you are concerned about any of your  processes, please do get in touch for a confidential chat.


Confirmation bias at the speed of light

The extraordinary world of the trader really opened my eyes

Image result for halo effect

Most readers will already be aware of confirmation bias and will no doubt believe they have it under control. Good for you. If however you are trader, or gambler the likelihood is that you have got it down to a fine art and it has become a heuristic behaviour.  What I mean by this simple, is that you’ve been doing for so long and become so attached to the theory that you are smarter than the rest, that now your brain does it in auto-mode without your intervention or even awareness.

Recently I spent some time developing trading tools for people earn a  living trading movements on a betting exchange.  The betting exchange, for those who are not familiar is a simplified clone of a stock exchange and is very similar to currency trading .

Some of these people act as bookmakers accepting bets form the public, while others simply trade the movements in the market or “SCALP” the market.
I was amazed to hear bookmakers of some standing use the phrase, “ Let’s get this one beaten”.
Quietly testing the intent of this over a period of time I realised that they actually associated their attempts to attract money and lay the runner with the runner losing.  Naturally they would never admit to it openly and they all know that such a thing is impossible, yet they daily select a weak runner and go about betting against it with the hope that it will be beaten and regularly it loses and bit by bit the brain has begun to associate tis intent and action with the outcome. Given they will be laying an 8: shot that really as a 1 in 12 chance of winning, the see it lose 11 times out of 12.
What is happening here is the brains own “Inspect and adapt learning process”, taking what it sees at face value and jumping to dangerous conclusions.
For years renowned physicians dispensed useless or even dangerous treatments to unsuspecting patients while convincing themselves and the patients that it was having a positive effect. After all, some of these people recovered. I wonder how much of this still goes on?
Economist and ex trader Max Keiser recently did a TV show on economics where he talked about stock traders believing they were changing the economy by their actions when in fact they are simply gambling with liquidity against other traders.
The UK government has been conspicuous in their inactivity in terms of fixing the economy, or even banking since the crash. They have sat back and conserved the status quo for all intents and purposes, yet George Osborne makes speeches in which he associates his office with a “reported “ improvement in the economy. Has it improved? If so, has it improved beyond a possible “ gently rising tide”?  Does he actually believe it?  What do you think?

Given the power of such self-delusion, would it be a shock if we found traders fixing things like exchange rates, or would it be far-fetched to imagine it might then stretch to bribing politicians and officials.
If those same people had gone into the munitions business instead of banking, do you think they might find a way to start wars? Do PMs and  generals ever admit that they achieved nothing ( best case scenario) or created a catastrophe?

Systems thinking

Negative bias the damage it can cause

Two important rules of the learning organisation that you won’t study in an MBA yet they are ignored at great cost.

The learning cycles described by    Sekar Sethuraman CISSP,CISA,CISM,CGEIT,CIA,PMP is in my view an excellent starting point that would most likely be a significant achievement to formalise and encode into almost any organisation.

My personal experiences apart from academic interest are around :

  1. What we often call “lessons learned” i.e. the constant adjustment to circumstances and to the perceived results of our previous actions.
  2. The impact of swarm intelligence on our ability to learn and teach.

Lessons learned, maybe we should not be doing it at all?

In the area of Project Management, few organisations do anything at all about “Lessons learned”  though virtually all would express a regret about this. In truth , doing nothing formally is not doing nothing after all and hence informal learning continues. Is that better, or worse? Well it’s not a clear-cut answer.

In the past year I returned for a period to the area of managing risk in an uncertain and volatile environment with vague rules and little explicit information.  A classic example of this environment is day trading , or any kind of investment banking activity, bookmaking, military activity, espionage, football  and a long list of less glamorous situations.  Football is too tricky for this discussion  because  there is almost no conscious decision making involved.
The reason I choose to discuss these more obviously volatile environments is because they are more like real life only sped up enough to trigger human emotions and  for people to  learn from trends  and responses  that otherwise  might remain hidden. i.e. because you see the results of your actions soon enough to make an association you have a chance to reflect on the actions and the outcomes that in normal, snail’s pace life , would remain hidden from most people .negative false

Actions and results are two key elements of all learning. John  Boyds OODA loop is a wonderful example of this.  What Boyd recognised is the need for “ Sense making” and this is the key, because learning without doing this effectively is to learn things that are patently wrong. The equivalent Is to put arsenic in your coffee cup.
In a fast moving environment we see every day people who pushed  a button a and felt a shock up their leg. Like the pigeon learning to select the right beans, he stops pushing the button, because he assumes a relationship between the two. How quickly he makes this assumption and how rapidly he reacts is directly related to his self confidence and very quickly you can see the cookie crumble to a pile of dust.  Burned out traders are almost as common as arrogant and broke ex-traders.
The answer lies in the ability to ignore what you hoped or expected to see, question what you do see and only act on proven information while filing the rest away for another day. The ability to do this is much scarcer than you might think.

Lessons learned can be formally handled in a project management environment and these lessons ingrained in culture at which point the will become pervasive until they need to be superseded.

That leads us neatly to the other area:

Swarm Intelligence or Hive Mind has most of us  firmly in her grasp.

Swarm Intelligence , or Hive Mind as I prefer, or in simpler terms culture is a far more pervasive and more potentially damaging force than most observers realise , in particular when it comes to learning.  Ask Paddy Power Bookmaers.- Paddy Power Left ‘Red Faced’ After Early Payout on Greek Vote. They trusted the wisdom of crowds and learned an expensive lesson.

Surowiecki had a bestseller and started a wave of books that appeared to discover something new in old wisdom , only to be widely discredited later.

According to Jesse St Charles of University of Tennessee at Chattanoogai, there are specific rules that define a swarm or flock:
1. The rule of separation. Think of a flock of birds flying in close formation but never  make contact physically. There is an unwritten rule that keeps them a certain distance apart and that rule alone defines where they go. Watch the starlings over Rome about this time of year.

  1. Cohesion. The birds all use the same patterns of flight and movement and even squeak and defecate in unison.
  2. Alignment. They gauge their direction by where everyone is going and align themselves
    4. Type recognition. A flock of starlings will never allow a crow to join, nor will he try

Once these rules are in place, the bird has waived all control over his own brain and simply follows the pack.  In all group based creatures this can be seen and it mostly likely stems for the safety of being in a group and ideally close to the centre.

Parallel this to the Stanford Prison experiment when a group of well bred and highly intelligent students form the top 5% or so of Americans were given roles and group structure in two opposing groups; Prisoners and warders and left to enact this for the benefit of a study.  If you don’t know what happened, I urge you o read this:

I am not commenting about the scourge of starlings in European cities, or the capabilities of the human given the right opportunity, I am simply pointing out that once an individual has identified his or herself as belonging to a particular group a large part o his/her brain is surrendered to the perceived group intelligence and the power of the written or unwritten rules of that group prevent learning anything that is contradicts the group in any small way


If you are engaged in deepening the learning of your organisation, or team bear in mind two extra rules:

  1. Unless you adjust the culture of your hive so that learning and changing is a source of social acceptance and security, all you efforts will come to nothing.
  2. Sense making, for adults in a business means something different than in teaching and learning. People’s emotions play a huge role in how they perceive the effects of their actions, what they learn from what they see and even what they see. If you want to create a learning organisation, you must teach and assist people to collect and observe the results of their actions in an objective way, make and execute good decisions at the right time and police their emotions against rash reactions, or unearned self-doubt.


Perhaps the short description of all this is leadership, the thing mankind craves for more than anything


How much damage can negative BIAS do to performance and quality of life?

Whether a manager, a sports person, an investor, or a dustman It is vital to understand our personal bias and manage it .

Every decision we make requires some level of objectivity to stand a good chance of serving us well.
“You’re faced with around 11 million pieces of information at any given moment, according to Timothy Wilson, professor of psychology at the University of Virginia and author of the book Strangers to Ourselves: Discovering the Adaptive Unconscious. The brain can only process about 40 of those bits of information and so it creates shortcuts and uses past knowledge to make assumptions”
 A large part of each decision will utilise principals and strategies that we have developed over time. Without these assets we could not function, our thinking would be just too slow
It is nevertheless also the case that alongside of these positive assets we  develop bias that is not helpful or positive.
e.g. an Employer with race or age bias will miss recruitment opportunities resulting in lost production and opportunities. Its probably impossible to be human and not collect some of this negative bias, but it is possible and absolutely crucial to know what your bias is and be vigilant. Thats a different thing from beating yourself up over it. You’re human, you’re allowed to be imperfect, but if you deliberately fail to do what you know is the right thing, that is another situation altogether.
– A footballer who always gives up at the last millisecond against bigger players and limps off apparently injured with a “losers limp”.
– A poker player who gets over ambitious every time he wins and then loses big time.
– A recruiter who dumps CVs with 10pt or smaller type and misses some of the best talent.
– A golfer who panics at the top of the back swing grips and swings down with all his force running out of energy, line and balance long before it gets to the ball and will often continue to do this for his entire lifetime.
A belief that bigger people will be stronger, or more aggressive is a common misconception.
Believing in the god of luck instead of the mathematics affects all kinds of people, even statisticians can fall to it.
Being too lazy to adjust the type size before printing is unbelievably common, as is struggling with weak eyesight without seeking help.
Focusing on the map instead of the territory or vice versa is universal.
These are very easy problems to cure, but some can be a little tougher.
Here’s the simple map: 
1. Recognise and admit that you do it and the cost to you, or spot opportunities to leverage it in your favour.
2. Work out how to confront your negative bias head on.
3. Recognise any influences that work to maintain the bias.
4. Keep on until the habit is replaced with the right behaviour.
5, If you can’t get rid of it entirely then simply keep checking for it and adjusting.
6. Play to your strengths and expose yourself to the situations you are best at.
How to watch for and recognize personal bias.
Look at the things that went wrong recently, track down the bad decisions you made that lead to them. Be brutally honest and replay the decision process to see where it went wrong. There, lurking in the shadows, you will often find a negative bias driving the bad decisions.
Another useful tactic is to ask close friends or colleagues that you trust. Be prepared to be shocked.
Finally look for the classic forms of bias:
Confirmation bias: Only looking where you can be confident of finding the thing you want to find. Or seeing what you expect to see in spite of the evidence.
Anchoring: My only tool is a hammer therefore every problem looks like a nail.
Halo effect: This works in football so its bound to work in the kitchen, or I am good at A so I m bound to be good at B.
Overconfidence: I am so good, I cant be wrong, don’t question me, or advise me. If I accept help, I will look weak.
Groupthink:  It’s important to fit in with the others, especially the cool ones.
Outdated:    Clinging to principals that once were sound, but are now damaging. (That includes situations when you are right)
How to confront personal bias head on.
It may not be easy, because the driver behind a bias will often be an emotion such as  irrational fear, hatred, greed, etc. Fear is by far the most common of these emotions. The other very strong candidate is peer pressure i.e.  you have this bias because all the members of your peer group have it and it is part of the acceptance criteria. Sometimes it is conditioned from an early age.
How to identify the influences and counter them
The footballer in our example could join a boxing club and realise that he can easily hold his own against much bigger guys, but smaller ones can be very good. That would be a way to get rid of the irrational fear and open the door for a more confident player that went the last yard with full commitment.
A sound strategy that counters many forms of bias is this; when you have time to do it, seek a conflicting view to yours and study it. Either it will strengthen your conviction or it will open your eyes. In either case you will reap big benefits. An easy way to maximise this affect is working with your team to arrive at joint decisions and acting as chair. This externalsies the process for you and develops your team at the same team.
Ask yourself how friends and colleagues in your circles will react to your changed behaviour. Often you will recognise the source of your bias as a peer group who reinforce the bias. This can be a tough decision, but sometimes one of them has to go.
Often one of these bias’ will seem to be part of who you are and you are reluctant to change, but that is simply untrue. Bias changes over time all on its own without your intervention except not always in a positive and helpful way

How to keep at it until you win
Don’t expect miracles, you will slip a few times, but focus your thoughts on the positive outcomes you are expecting to achieve. The moment a negative thought about it begins in your mind, immediately replace it with something pleasant, or even downright naughty, but zap that negativity. When you make a mistake, pretend you have just had a triumph, celebrate and reward yourself so you stay positive. If you can find an admirable role model that will help an awful lot.
Once the benefits begin to accrue it will become a no-brainer.
How to be vigilant
Lets say I’m an investment manager and I handle billions in investors money. I have somebody data mining my trades to point out to me when a bias appears to be forming. He rushes to my desk and he says: “You have sold a day too soon on 70% of hour trades since last Monday, is there something you need to talk about?”
I think it over for a while, we look through the figures and I see that I had a heavy loss a week ago when I held on too long. I had been warned about the potential effects of a certain political situation and blamed the error on being too optimistic. I have been running scared  since and selling in panic.
Poker payers call this “On tilt”, For most of us, it is unfortunately part of our untutored makeup, but it is deadly in the world of a dealer and won’t do the rest of us any good either.
Once I know it exists and I can see the cause, fixing this is easy. I just watch the charts, follow the rules, feel the fear and deal with it.
The real damage that negative bias does is rarely mentioned
To fully understand how important this  subject is, you heed to delve a little into how the brain works.
Learning is a process of:
1. Observing our environment and absorbing communications and stimuli,
2. Making sense of the information we find is a process of comparing new information to stuff we already have stored away and passing it through filters to find out how we will classify it. These filters are made up of strategies and principals (positive bias) and negative bias. These are applied in the “fast thinking” subconscious part of the process. Consciously considering it all and applying logic.
3. Deciding how to classify the new information and whether we want to keep it?
4. Acting on your new knowledge means using it in the real world and this  is the final step in learning.  The problem here is that unless you are capable of honestly and objectively examining the feedback, your learning process is already broken and you are destined to swamp your brain with destructive conflicting rubbish.
How you classify information will govern whether or not you are able to reuse it. If you are carrying around filters that are wrong then you will spend your time rejecting valuable information and misclassifying critical knowledge. The damage continues and escalates.
Finally, when you act upon your new information and look for feedback, you will accept stuff according to your broken filters and the other broken bias such as conformation bias, group think and their mischevious friends.

Managing our personal bias is as important as brushing our teeth, but the rewards are much greater. It’s not about not making mistakes, it’s about watching out for them and taking action.
A video made by Google is revealing.
Word or concept associations in dictionary form is a key part of how humans store and retrieve information.
The speed at which we are able to memorise these associations  is far superior to any other form of information storage and retrieval.
Speak the name of a country and most of us can instantly reply with its capital city. It comes naturally.
When we make an association between grey hair and age or age and frailty, or obesity and laziness etc etc, we are then sowing the seeds of bias that is neither useful nor  helpful.
Take the controversial IAT test to help you discover your bias.

Using personalisation cleverly to grow your customer relationship and keep your sanity

In my next  instalment I plan to look a little more closely at the broader context of recommendation engines and how they might be used but foe now I will simply point out that they are broken into two broad types Content Based and  Collaboration Based. The first refers to product knowledge and the second to behaviour of customers. Today I am only discussing the latter type

There’s two fundamental types of customer information used in recommendation engines and customisation strategies: 1. Real time knowledge of customer in context and 2. Historical knowledge of people with context.

I like to refer to them as 1. Personal and 2. Collective.

The first problem you run into when you start an ecommerce store is that you don’t have any historical data to start from. Making decisions about “people” is another way of describing prejudice. Of course all prejudice is not bad, we help old ladies across the road and we are kind to children. But when somebody hears my accent and offers me a particular brand, I am irritated.

What I find especially infuriating is when I simply cant find what I want on Google because it thinks it knows what I really want based on the fact that I looked for it twice last week.

The point I am making is that people are individuals, they change their minds, they like variety, they find what they are looking for and loose interest, in fact they really do not want to see what they just bought on offer at a better price.

The bottom line here is that relationships have always been difficult even when you are at the next desk or sharing a home, let alone seeing a cookie form the same machine create different click streams and view different content.

Personalisation and making recommendations based on algorithms can be high risk and must be done with care and forethought.



Context is a critical factor in storing and classifying interactions for recommendations and personalisation.  When I go in our local store the shopkeeper who knows me very well takes into account the day of the week, time of day, look on my face, which shelves I’m looking on and what he remembers about my usual purchases and he then decides whether to let me walk out or to ask if he can help. He rarely gets it wrong.

If he did get it wrong very many times, I would be very tempted to walk a few yards extra and shop anonymously. Why should I have to explain myself?


Imagine if he wiped lots of stuff off the shelves on the basis that I won’t be interested in them today, but filled the shelves with what I bought on this day last week, or on my last visit.  Not very good.

Imagine if I run a social network, I know a great deal about where you are going, what you like etc. If I provide your phone network, I know who you call, and receive calls form, when you use it and how long for, when you never use it and when it is off and where you are right now. Maybe I see your calendar and your emails too? Not to mention half a dozen spook outfits eavesdropping.

With this kind of information I could be very invasive, so it is critical to only use this type of data with the customer’s consent and to put that customer firmly in control in case they should change their minds. The way to achieve this is to offer them value in return for their data.


In a typical e-commerce implementation you will have access to the following types of information:

Purchases,  wish list adds, cart adds, views, shares, favourites, downloads, shares, watches, likes, follows, where they were referred from and you may know about the search terms they used to get to your site.

You may also know about the sites these customers visited after they left your site when they have been active, their spending patterns, whether they buy for children, their gender, devices used, devices ordered from, devices emails were read on, where they have browsed from geographically, and a whole lot more. Once registered you could collect data on their social media activity.

If you are an OmniChannel retailer, you will have more data in your CRM, your ERP, and other places

data sources

This information exists in many places: In your customer database, in the web server log files, out there on social media sites, on Google, in an advertising network you work with,


In a typical solution, the information might be classified in three ways: 1. Customer , 2. info and context 3. Offer.

e.g. you create facts such as:

Customer, info,  (info context), Offer.

John Brown, likes(3) (June 4 14, Saturday, mobile ), gizmo product

John brown, price search(June 6 14, Monday, PC), gizmo product


Each piece of information is about a specific offering or product and it relates, of course, to an individual customer. In addition to that basic information, it is invaluable to collect context.

Understanding that Mr Brown never orders on his mobile, but does look at info helps you decide what to present to him. Maybe the solution is to offer him a convenient app that makes ordering easier. Certainly if 80% of his browsing was on mobile devices and 100% of ordering was on a PC,, this would be a strong indication

The way you might arrive at this type of conclusion would be to query these stores of facts, summarise what we know and make inferences.

This is a lot like data warehouses where you store vast flat tables, analyse them once then stores summaries in a more accessible system.


e.g. John Brown, likes(3), gizmo product  That weighted his interest at 3

Two days later he returned infer a higher interest, say 4

He asked for pricing info, this puts him at 8 or thereabouts

So we can summarise it as ; John Brown likes (June 6 14),gizmo product (8)

We know that he orders via PC, that was not a barrier.

We know that seeking price info is a sign of desire and conviction so why did he not order?

We might assume the price was too high.  Perhaps we are all too aware that we are not the cheapest in this market and we rely on positioning to overcome price objections.

We may decide to send a once off special offer on Thursday morning, so he cant resist because he frequently browses then so he may have free time on Thursdays.

That is an example of generating a “Next action” on the basis of knowledge and intelligent inference.

Of course it could be that he planned to order on Friday at the full price, that is a price we either do or do not decide to pay for John Brown’s business. Eventually we will know enough to be able to predict this accurately.

If we knew that he was a highly influential blogger with a large following, I wonder would this influence our decision?.


Contrary to what is written by cloud vendors and Hadoop junkies, this kind of data and that kind of information is rarely dependent on petabytes of data spread across numerous clusters. It could be done with a basic RDBMS, though it would probably be easier with an RDF store downloaded for free.

Don’t get me wrong, there would still be some development to do and it wouldn’t work till a bit of data had built up, but you don’t have to be Google or Yahoo to benefit from this technique.


An interesting side effect of this approach is that the longer the customer keeps visiting the bigger investment you have in keeping her, because you know exactly how to twist her arm. Her customer lifetime value is infinitely greater even at a discount. Perhaps eventually, these relations will be reflected in the price each individual customer is asked to pay.  There’s a thought to conjure with.

What if John turns up at your Branch in Oxford street with his decision made and ready to order? What if he orders online but wants to collect from your store in Paris next week?  What if there’s an issue after delivery and he wants to get help form your store in Las Vegas while on his travels?

What if the cart tells him “sorry, we’re out of stock” and then he walks to the shops and sees one in your local store?

I know just how difficult these problems can be and how expensive it is to overcome them. It requires not just a technology rebirth, but a culture change like no other.

Project managers must be consultants and must behave like consultants – it’s a bad day for “permies

Ed Taaffe has enjoyed a twenty year career in management with the past ten years spent in Project management and takes a special interest I human motivation as a management tool.  In this piece he explains why the consultant relationship with the client and the project team is critical to the success of projects.

This argument has nothing at all to do with the perennial  disagreements between Contractors and “Permies” in the IT industry, though it does go part of the way to proving the contractor viewpoint right.

What does success look like?
First let me be specific about the project failures we are addressing. There are all kinds out there and as any good problem solver or six sigma practitioner will tell you, there is more than one way to apportion blame and define cause.

What I am specifically addressing  is the high proportion of projects that get closed down,  or arrive late enough , or sufficiently over budget, or of sufficiently low quality to be deemed a failure.

These projects all started off wit everything in place and going fine, then at some point they began to drift and continued to drift unchecked until they ended up a failure.

My assumption is that since many projects defy totally accurate prediction of time and cost, there is a built in expectation that budget, scope and or duration may have to move at some point, if this is done and agreed then there is no project failure. Project failure occurs when the rot sets in and it is ignored, or when the project is set up with immovable boundaries and finishes outside of them.

Knowledge management holds the key

Within the technology world there is a particular inability to understand the nature of knowledge and it is mostly equated to data, or at best and not often, to information. The trouble is that neither of theses viewpoints is helpful.
 It is OK to believe that a  software process demands a precise piece of data to work correctly and to live in a virtual world of absolutes as do many technically minded people, but that doesn’t wash in the real world and there lies the corpse of many a CIO.

in the real world of doers, makers and shakers, decisions are made by people and the key ingredient is not the data, but the implicit and tacit elements of the way the people interpret information.

Bear with me, I’m almost done with the boring stuff.
Cognitive Dissonance

Leon Festinger produced a study in 1957 at Stanford University whereby he clearly demonstrated that when somebody has reason to present an argument that is actually at odds with what he believes, he naturally alters is opinions as a result and finds himself agreeing much more with the argument that he previously disagreed with strongly.

It’s not hard to imagine how and why this might occur and there is plenty of further work exploring this aspect of the phenomenon of Cognitive Dissonance, however the most interesting part of the experiment carried out by Festinger demonstrated that when a reward, or threat was used to force, or induce the person to argue against his own beliefs or judgement, then the effect of Cognitive dissonance was lessened, or missing altogether.

Clearly the mind has no trouble in understanding the idea of being paid to hold an entirely objective opinion, which it seems almost incapable of achieving under other circumstances.

Implicit and tacit knowledge

Interpretation of information and comparing it with learned responses and experiential knowledge and bias is the essence of implicit and tacit knowledge. It is therefore critical, naturally that the information is interpreted in a fairly objective way if the resulting knowledge is to be accurate and reliable and good decisions made.

Take the situation where the project manager has been indoctrinated and reaffirmed again and again that the project is on schedule and will not fail, or will not slide and he has sent out the RAG reports and made reports and presentations to stakeholders and boards convincing them that everything is going well, how do you expect he will react to data, or information telling him that several key tasks have slipped and there are issues looming?

It’s all in state of mind

The fact is that if our project manage is part of the culture and one of the pack and he feels the pressure to make this project a success, he will convince himself so strongly of this that he will behave exactly like Festinger’s students did in his experiment back in 1957, he will fail to see, or assimilate that which contradicts what he has been told to believe by the peer group, that which is contradictory to the crowd consciousness.

The answer is simple

The project manager must be an independent consultant and he must be a facilitator only.
He must have no personal stake in the success or failure of the project in terms of hitting dates, amounts, or quality targets, but he must be someone who talks straight, keeps the “permies” honest, ignores the crowd pull and tells the Empror when he is wearing no clothes.

The five stages of an IT project

1. Enthusiasm for the goals
2. Disillusionment with the progress
3. Search for the guilty
4. Persecution of the innocent
5. Praise for the nonparticipants

Einsteins definition of madness was “someone who keeps doing the same thng and expecting a different outcome.”

Johari’s theory defines four aspects of knowledge and ignorance.

Knowing what you don’t know,

knowing what you know,

What you don’t know you know

What you dont know you don’t know.

The third is a terrible waste and the fourth will make a monkey of you every time.

Wise up and get some training, get a coach, or get someone else to do it.


Ed Taaffe






Reblog this post [with Zemanta]

Tim Berners Lee is on the money yet again.

There’s been some hype going around that the internet volumes are beginning to clog it’s arteries and that there are issues ahead.

TBL reacted to this by saying something to the effect that; the internet is not at risk form volume of information but volume of misinformation.

Let’s just think about that. There was a time, long before Google, when a tool called webcrawler claimed to index every document on the internet and as I recall, it went damned close. Today nothing can do any better than scrape the surface and what is more, there is little appetite for doing it.

It was incredible what you could find with a bit of patience back then. For me the quality lay in the surprise. I was exploring as opposed to searching. In those days surfing was something a few cool people did instead of watching Corrie. Nobody has time for surfing now though.

The novelty became the business tool, became the competitive advantage became the essential productivity tool in less than a decade. Not only do we not have time to explore for nuggets of knowledge and correspond with interesting knowledgeable people around the globe, we need filters to delete 80 % plus of our mail, we are trying to develop trusted networks to find people it is safe to talk to and our problem in finding the information we once only dreamed of, but now rely on for survival, is one of wading through the rubbish to get to it. I suppose this effort counts in that equation, but if you can’t beat them, well!
My point is this, if you don’t know something, or worse still you don’t know that you don’t know, then more information, data, or even knowledge is probably not going to help you and in fact, it is inevitable that before long you will find several contradictory experts.

So where did it go wrong?

Man’s access to and use of knowledge has always involved trusted sources, be they books, or advisers and when things were really important, second opinions were critical.
It was never important to be right, just to be trusted. All sources were expected to be wrong sometimes, but far better than no knowledge at all

My favourite description of knowledge is the one that says it is “to know”. That implies it is known by a person. In scientific terms it is often broken down into explicit, implicit, and tacit.

Let’s explore this concept for a moment;

Very little knowledge is explicit and it generally refers to hard data in my view, though some will argue otherwise.

The fact that john selected b as his favourite response out of abcd is explicit. It’s not much use though.
Let’s jsut assume that we all now what the context of absd was, 100 replies like this would make up a useful block of explicit knowledge.(data in my book)
What that information tells us (implicit knowledge) , may be that the males prefer B and the females D. However, an experienced researcher might spot the fact that it had nothing to do with gender but, was a result of colour blindness, or another aspect lost on the rest of us. (tacit knowledge)
At this point you have the full package and you are reasonably safe to make a decision of some sort.
My view and most business people would go along with me, that any less than the full pcakkage would be dangerous indeed.

If we follow that hypothesis and set these standards for knowledge and if we measure the internet in this way, there is an enormous need for reliable knowledge that combines at least the first two elements if not all three, in order to be truly useful as a source of knowledge.

So far we have succeeded in creating a few working proptypes of semantic modelling and inference engines that are capable of implying a level of information int the raw data out there, if and when we ever find a way to implement it.

Here are my questions:
 Where does the internet serve this need for knowledge?

Is it ever likely to get be achieved?

Is this need actually worth serving?

What would success look like?

Would more information or even more knowledge lead to better decisions with any degree of consistency?

Ed Taaffe