Big data rules but, Netflix’s Secret Special Algorithm is a Human

hawaii kane

Imagine the scene.
The great and the good of the Polynesians gather in front of the cave where dwells Kane the great god of wealth and power eager for a sign of some sort that will guide them in their quest.
In the cave is the devious son of the Islands only coconut farmer and with a deep, practiced voice he says, “Eat coconuts every day and you will be rich beyond your wildest dreams”.  Nobody in the audience hears or sees anything, but what they desperately want to hear and see, nor do they want the trouble of a bit of thinking and deciding and  soon they all drop to their knees in worship and the party goes on until dawn.

Guess who got rich and powerful beyond their wildest dreams?

In this blog,  I talked about the powerer of personalisation and went to some depth about the possibilities and limitations and specifically we discussed one of the acclaimed  leaders in personalisation, Netflix. Most people are familiar with them and believe they do a fair job of making recommendations.

Imagine my chuckles and many peoples amazement to read that;  “Netflix’s Secret Special Algorithm Is a Human

Yes, Netflix has used their customer insight to fund a movie and the movie  was watched by lots of people (duh!) and they made this clever decision that was watched by so many people because of the remarkable data insight they have.

No, not quite!   Here’s what happened.
Netflix asked an individual who has probably demonstrated this ability to them in the past to decide what their customers would indeed watch.  Having funded the movie, they then threw the might of all their considerable marketing ability and access to their own customers into making sure they did indeed watch and talk about the movie.

 polynesian big data  conference

DataCon 2015

It’s one thing knowing a few things about how to make pieces of data comparable and physically compatible and even knowing how to get it out of all the competing pieces of technology and into one place and occasionally someone may even be able to preserve scraps of the context in sort of usable form, but it’s quite another to assume that this explicit data is knowledge let alone intelligence.

Thee are many clever people working on development of tis higher level of machine intelligence and with a very long way to go. In the meantime it does no harm to classify knowledge in tis kind of fashion:

Explicit – mostly just data and not fit for purpose

Implicit – Enriched by comparison to other knowledge that supports developing good hypothesis.

Tacit –  Enriched by the experiential knowledge of people and experience and ready for use with caution and consultation. Clearly this is what Netflix have done  quite successfully and hopefully others will learn from.

Coconuts are rich and pleasant food and there’s plenty to be said for eating them, but there’s a limit. Clever collection use and interpretation of data is a powerful thing and nobody should be ignoring it, but I have to laugh out loud at some of the stuff that is written, often by people who have recently graduated form a typewriter.

For further reading see:

Using personalisation cleverly to grow your customer relationship and keep your sanity

Are you wondering if a recommendation engine is the next big purchase for you?

Give big data the heave-ho for now and get a birds-eye view of your business without a single nosql database.  

About the author

Stage gates are still imortant even in Agile product maanegement

When I was a developer in the late (90s, I had the benefit of having started my career in marketing and combined with an overdose of enthusiasm and self-confidence, I thought I knew better than anyone about how the product should be designed.

Not that many years later I was in charge of Product development and planning and with the benefit of some training and mentoring from some of the best, I was acutely aware of my frailty when it came to predicting the present, let alone the future for long enough to make a business case deliver and keep a whole building full of us in employment.

The fact of the matter is that because there is huge demand today does not mean there is means,  nor does it mean there will be demand, nor means next year. A market opportunity is a different can of works than a product opportunity and the first stage gate must at the very least ask the questions; Is there a strong possibility that this opportunity will deliver the benefit’s we need? Are we able to do it? Is it right for the brand or will it confuse our customers about who we are? what impact will it have on other products we have that may be rising stars or in the long tail generating strong revenue?

Of course you probably have a few more in mind, but those few alone are sufficient to point out that there is serious strategic thinking to be done before you scribble an elevator pitch on the wall of the developers den and go off to lunch.

As we progress through a software development project, we often come up with new problems we had not identified, or problems that are simply a whole lot bigger than expected. They may not always be technical, but could be regulatory changes  etc.

When these things happen and the likelihood of achieving an acceptable offering and still being able to return a profit is threatened, then there must be a stage gate process.

When the early launches are not achieving the type of results we anticipated and we don’t know why, there must surely be a strong argument for stepping back from it and taking an informed and sober view of the chances for success and the opportunity cost and potential reputation cost of a failure. Of course there is always an opportunity to handle these issues on the fly and do it fairly well, but there are times when a more sober and considered approach is a good idea.




How to implement Scrum while keeping your sanity and your Job

Cost overruns, missed schedules and quality issues. These were the reasons cited for the development of Scrum.
Management also wants to harvest the business value expected from new software sooner, while using fewer resources

Breaking a traditional process up into small increments solves all of these problems. MSFis a wonderful example of this apart from the fewer resources bit, that is a myth. MSF encourages the business to develop a long term vision for the solution, but then it is broken into a series of small deliverables that build up in timeboxes to a full solution, or stop when time or budget run out.

The benefits are significant because; a. A proper Enterprise Architecture, style and vision is agreed first. That means that we don’t start off with one approach then half way through have to scrap it all and start again in order to scale. B. It means that at the end of the first timebox there is something of business value to show to the customer.

MSF differs from scrum however, in that it still depends on some basic documentation and designs and formal testing in a way that is familiar to most software engineers. MSF also lends itself well to procuring systems which I can’t honestly say about Scrum.

The downside is that MSF does not greatly improve communication other than the fact that it keeps the timeboxes short and encourages a collaborative agile team of peers, but the documentation still poses a threat to communication.

Scrum is a very different kettle of fish.
Scrum is designed to address a number of other problems:

  1. The Product Owner is a response to the lack of communication in traditional projects caused by remote stakeholders and by the sheer challenge of communicating a business idea from an executive to a programmer or architect accurately. Putting the business person in the role of Product Owner broke through this barrier.
  2. Demotivated and remote “techies” interested only in playing with gadgets is also a perception that still exists and it is countered by empowering them, making them into generalists and putting them in daily communication with their customer in the business.
  3. Fear of not knowing what “they are doing in there” is still an irrational fear that plagues the relationship between technology and the business. Delivering demonstrable pieces of functionality at regular intervals lets the business see proof that they are working and usually sends them off to sleep within a few months.

Aspects of Scrum that pose serious threats to success of a project

  1. Scrum was designed for writing code for fairly small projects. It was never designed to be used for many of the other unbelievable uses it is put to. The reason for some of these ridiculous applications of Scrum may be that Project management skills have been deprecated in favour of” the new thing”. The chances are they were not great to begin with and hence the adaption of Scrum
  2. Scrum is NOT Project Management, it’s designed by software engineers for software development. In small projects, the Product Owner could carry in his head all the stuff normally entrusted to a project and the scrum master can run the development, but this does not translate to bigger projects.
  3. Scope creep. Yes, we thought we had beat this one but in fact the reality is often that Scrum introduces a crippling level of scope creep. Three reasons mainly are to blame:
    1. No formal change control, nobody to say ” you are kidding?” so everything goes on the backlog
    2. No formal end date, so as long as it is all cosy, the business customer will keep on going, why should he stop?
    3. Bugs and millions more of them. Scrum teams don’t do formal testing and if you ever developed software you know what this means . Scrum teams also decide which bugs to fix and which to ignore. Naturally this becomes a tool when meeting the end of sprint deadline, it may be that the system works, but only just. That is acceptable by the way.
  4. A Scrum of Scrums is like replacing a big house with a tower block. It isn’t smaller or more manageable, it’s a mess. We spent the last twenty years turning old fashioned command and control organisations into flat collaborative ones. Now the scalable Scrum has built them back up again with a cell of cells like a medieval army. I’m not saying its impossible, just that it needs improvement and it’s not always worth the effort.
  5. Estimating is very difficult and Sprints are far worse at predicting their time and scope than traditional projects ever were. Costs are sometimes spiralling rather than falling, they just are not visibly so.
  6. Frequently the techies dominate the Product owner to get their own way and the collaboration is little more than a nice theory, the difference is he won’t tell anyone.
  7. Scrum was designed to be done by very experienced engineers all of whom were trained in Scrum. In reality few team members are even certified and the engineering skills are generally not good enough for a self-contained empowered team
  8. If any of the team members leave it can have a serious effect on the outcome.
  9. Scrumbut is a major problem. It refers to the practice of leaving out key parts of the process because you don’t like them, or whatever excuse. The problem is that velocity charts, retrospect and all the other rituals are there for a very good reason and when left out you end up with a traditional bunch of techies running amok with no controls, planning, or performance improvement strategy.
  10. The hatred of documents and authority figures really hurts when it comes to architecture.
    The team gradually becomes braver and makes more unqualified assumptions about the underlying infrastructure they are building on until eventually disaster happens and when it does there is no documentation, the designer has left and there is chaos.


So what should do you need to do if you are considering a Scrum implementation in your organisation?

  1. Begin by spending as much time as necessary to be very clear about what problem you hope to resolve with Scrum, or what benefit you want to reap.
    Don’t implement Scrum to deliver major projects, it was never designed for this. If you want to add some of the elements of incremental delivery etc, go ahead and introduce them to your traditional approach, or explore another method such as MSF.
  2. Define your criteria for deciding whether a project should be approached in a traditional way or via Scrum. Get advice if needed and especially in the first year, be very careful to tackle only projects that lend themselves to Scrum. As a guideline I would say:
    1. They should have a small element of invisible work, i.e. if there’s two months work to implement stuff you can’t demonstrate, don’t tackle this one with Scrum at least not till you are very experienced.
    2. The solution must be something that lends itself to incremental delivery and can be meaningfully demonstrated.
    3. The whole team including Product Owner can be located in the same building and I would try to get the Product Owner relocated to sit with the team.
  3. Define a clear framework for Enterprise Architecture and let the Scrum team clearly understand what they can touch and what they can not and don’t budge on change control mechanisms when it comes to mission critical systems and those that impact them.
    In TOGAF terms a good approach might be to say that only the Organisation specific technology can be touched and within that I might label parts of it uniquely for agile access.
  4. Don’t start with a team who have let you down at traditional approaches and read an ebook on Scrum. Send them away for a few days to be trained and hire an experienced person to coach the team through at least their first project.
  5. Put in place a QA process to monitor Scrum and their performance. Make sure that all the rituals are being done and in a meaningful way. Make sure the velocity is being monitored, monitor the accuracy of estimations, watch out for debt in the bugs department.
  6. Most of all, when you have a good plan in place and you are satisfied that you have the answers to hand, present your plan to your colleagues in the business in the context of how it will be impacting them, how they will benefit and how they will be expected to change the way they engage. Make necessary changes and then plan for the launch.
  7. This is really important. Take a cold look at your existing project management processes and ask yourself what is it that made you so keen to try an alternative approach. If some of these issues are not going to be solved by Scrum and I predict that will be the case, then set about resolving them in another way.
  8. Make sure you have a clear interface defined for how Scrum teams will interact with traditional projects and Programmes when asked to do so. There must be no them and us, but on the contrary, the two disciplines must support each other. You will need your programme and project management capabilities for many things and it will be beneficial to be able to interface a Scrum project with part of a project or a programme without ruffling feathers. This will also save you from the ignominy of trying to deliver huge projects via Scrum


Intelligent use of Scrum to develop spirally

Bulls eye product management
Spiral product development with Scrum helps you hit bullseye and stay on target

I wrote previously about the Product Ownership in my blog Product Ownership the great let down in that blog I discussed the importance of stage gates as a safeguard when developing products for a crowded marketplace, especially new or very different products.

The MRD is typically the final output of a Product Manager exploring a new product idea and it sets out the product vision along with the estimated demand for such a product by way of unmet needs in the market. It will size that market and make estimates of break-even and the likelihood of achieving it and going on to establish a healthy product lifespan. At the end it will make a recommendation. The stage gate will consist of a review when product, technical, financial and other strategic stakeholders will consider this information in the context of their own knowledge and risk appetite and a joint decision will be made to proceed or not. This not very different to the way an internal project might begin other than the type of information being considered and the type of risk and opportunity profiles involved. I find it very hard to visualise a situation where this work should not be done.

The risk are simply too great. Changes happen daily in global markets, finance, economics, local markets, disruptive entrants and so many other potential parameters that without serious initial and ongoing assessment the likelihood of burning large amounts of capital regularly is just too great.

Most product organisations now and in the past would have a series of stage gates which allowed them to develop the proposition in increments and then reconsider it based on the new information. For many, the extra security offered by this method is outweighed by the level of formality and constraint.

For those very reasons, agile can provide excellent solutions to fast moving environments, changing information and uncertainty. Agile follows an incremental method of developing the proposition and the product in tandem. I often refer to it as the Onion Plan.

It begins with the product owner defining a Minimal Viable Product ( MVP) or Minimal Marketable Product ( MMP).

Marty Cagan gives my favourite definition of this in his blog when he says “I have long defined minimum viable product as the smallest possible product that has three critical characteristics: people choose to use it or buy it; people can figure out how to use it; and we can deliver it when we need it with the resources available – also known as valuable, usable and feasible.”

I have also often heard it defined as the minimum product that delivers on the product vision. This is also good but it misses the important point about being marketable. Otherwise people won’t get it and it can do just because of that. E.g. A three wheel car works fine, but its hard to get customers excited about it and their reactions say nothing about the four wheel version you are planning later in the year

This initial product forms the core of the onion, at least as you initially see it. That does not mean t forms the core of the product, because the whole point in getting it out there is to react to feed-back and make changes, so the core could be moving this way or that before and while building up the further layers of the onion. Needs versus features



When we define products we discover a need in the market, it represents a pain point for our customer for example. E.g. Customer tell us they need their cars to burn less fuel, so we respond by designing a car that that runs on half the amount of fuel. Then we discover that thy are not willing to live with the smaller size and they must have a minimum accelerations so we revise our product to save 10% of fuel without losing any comfort or acceleration. They like this, but now it is more expensive than a better one from Japan and they won’t pay our price so we go back to the drawing board. Eventually we reach the right compromise and release a simple model that has a broad appeal to our most important segment of the target audience. We are able to sell this to real customers paying with real money and get their feedback. That in my view is when the Onion core has been built and now the layers start to be added as we address the extra needs of our more sophisticated segments, watching at all times for a change to the core proposition through a new competitor offer or an unexpected macroeconomic change etc. Either the need or the potential solutions can change in our market environment at any time as can the macro environment such as their ability and willingness to pay and all can have can rapid and serious consequences for our offering.


The skill of the product Owner in this circumstance was to identify the potential core product, address the right segment, get good feedback and understand it then come back rapidly with a better offer and keep doing this till he hit bulls eye before building on extra layers of features or addressing other segments. Understanding a need is a particular skill, but meeting that need in a competitive way that can drive strong market performance is another entirely separate skill and both are inseparable from each other.

This does not sound at all like a bunch of techies huddling over a wall covered in scribbles, but it is in fact a very effective use of all the principals of Scrum as long as there is enlightened and capable Product Ownership.


The IT guy and the balloonist

“A man in a hot air balloon is lost. He sees a man on the ground and reduces height to speak to him.
“Excuse me, can you tell me where I am?”
“You’re in a hot air balloon hovering thirty feet above this field,” comes the reply.
“You must work in Information Technology,” says the balloonist.
“I do,” says the man, “How did you know?”
“Well,” says the balloonist, “Everything you told me is technically correct, but it’s no use to anyone.”
“You must be in business,” says the man.
“I am,” says the balloonist, “How did you know?”
“Well,” says the man, “You don’t know where you are, you don’t know where you’re going, but you expect me to be able to help. You’re in the same position you were before we met, but now it’s my fault.”
We have all been in that field or hovering above it, or even both at different times and I expect we can all sympathise immediately with at least one of these characters. The sobering thought in all of this however, is that the man in the field won’t survive very long unless the balloonist finds his way home and the balloonist stands little chance of surviving without the help of the man in the field.

The baloonist talks turkey

 Did I mention that your salary and bonus are in this baloon, but I won’t be able to deliver it said the balloonist, looking rather smug.

He went on, do you remember when you came to me for more budget so you could hire balloon analysts, to understand my needs and provide a better service, well that was when it became your problem, so maybe we should start again.

Can you tell me where I am?

The IT guy replies

Actually no, it became my problem when you got lost with my salary on board, but unfortunately, I am only able to understand how your balloon works and help make it work better, that’s what my analyst helps with, but deciding where you are, where you want to be and how to get there, is still your responsibility and incidentally, one that I am not qualified to advise on.

I ask, you tell, I provide the supporting technology.

By the way, I take it your salary is in there too?

Baloonist replies

 Yes, I believe we have had a misunderstanding and I see we have a shared problem.

You have the option to find a more reasonable employer and I wouldn’t blame you.
I on the other hand, have the option to outsource to someone who will shoulder full responsibility.   I had an Indian team here last month who know our industry intimately and are working with some of our biggest competitors. They will advise strategically, partner with our SMT and provide us with thought leaders and presentaions, provide all the business analysis and deliver the end results and they will do it at offshore rates.

What do you think we should do?

 The IT guy ponders and then replies.

 If I look for a new employer, there are indeed many options, but I expect all of them will be much like you or have other equally difficult issues, so I see the solution as more important than the geography.

 My gut feeling is that if you really believed in Santa Clause, we wouldn’t even be having this conversation because your Indian pals would already be sitting in my office.

Once you begin to outsource, getting out of that is the most painful and most expensive experiences you are ever likely to have. You will have mission critical systems that nobody understands, or can maintain and you will be forced to accept any amount of price rises or reduced service levels. The path to extricating and uncoupling your systems will look so impossible as to not even warrant consideration.
Controlled outsourcing may well bring benefits, but you will need me more than ever if you go down that route.

If you pay for the champagne I’ll happily take you and your other directors to the races, I will be more than pleased to keep you up to date on the changing face of IT and it’s implications for you, but I don’t believe it is appropriate for anybody but yourself to make strategic balloon decisions because they are ultimately your responsibility .


 Maybe we need to have a rethink and start putting the relationship on a more realistic footing

 It guy


Where has all the money gone? – you wouldn’t believe it

Part one – why?

Previous installments:
IT investment for the small businessman and novice
Why the SMB/SME holds all the aces when it comes to IT
In our last instalment we talked about the advantage an SME/SMB has over it’s bigger rivals when it comes to implementing technology, especially around the aspects of communication, business change and lack of red tape.
Don’t let this reference to red tape fool you. It is mission critical to understand the difference between vital  process and red tape and it’s not always immediately obvious.
The key to cost cutting in any area is knowing what to cut. The word we are after is waste, but even that can be misleading.  In the world of Lean everything outside of the critical value adding moment is deemed to be waste, but nobody imagines that you can eliminate most of it let alone all.
The best approach is to look at where the money goes in a typical systems implementation.
Let’s take ACME  cash registers.
They have a turnover of 200 million and they have countless systems handling different aspects of their relationship with customers.  Nobody in the organisation has the full picture at any time bout any customer.
Sales people call with an Upsell proposition within an hour of the customer complaining bitterly to support.

  • Customer enquiries get written on cards and handed to salespeople who then lose a percentage or forget to mark them up so 14% of leads are never followed through. (any of this sound familiar)
  • The last time they tried a mass emailing they only had email addresses for 34% and two thirds of those were returned undelivered. This resulted in their IP address being registered as a spam house and for three months their emails were being destroyed by a robot and never arriving.

Ok this is just the tip of the iceberg, but you get the picture. The new VP of sales has read the riot act and reluctantly he has been promised that they will try and find some budget to help out. Where do we go now?
In our next contribution we will provide an insight into how not to approach it and the we will move on to  look at a simple model for getting this job done.
Requirement gathering the first big mistake

  About the author


Get a bridger badge and publish your link

Get your Bridger badge and get a link to your web site or profile.

Background to the bridger group.

Recent research has returned a unanimous demand for a bigger effort from IT professionals to communicate with businesses, to serve business need and to become more user friendly.
While there are rights and wrongs and more to this than meets the eye, we can all benefit ourselves direclty and via successful clients and employers by demonstrating our commitment to these ideals.
Only by making our clients successful will we prosper and only by convincing them that we can be trusted to put their interests first will we be chosen to work with them.

The first step in this journey is to demonstrate publicly your commitment to these worthy principals by displaying the badge and promoting the principal of bridging as a way to reassure clients and employers to trust us, hire us and follow our advice.

Why get backilinks?

Marketing professionals spend vast sums annually on text adverts (URL Links back to their web sites). The reason is simply to please Google into placing your web site or profile close tot he top of the list when someone searches for a business analyst, change manager, or project manager and related terms. Google judges you by the company you keep. Expensive text ads are cheating, but free links for friends and colleagues in the business are the fastest way up the ratings, so start getting links now and I will be sending you more introductions as suitable profiles are added to the list.

Why be a bridger?

The one thing an independent consultant or a professional lacks when t comes to convincing a potential client or employer to meet with you is a strong brand. They just don’t know what you stand for and are nervous about taking a risk with you, especially if they can call a well known firm or get a recommendation form a trusted adviser or recruiter.

By publicly demonstrating your commitment to the stated needs of your potential client and being seen in the company of many more like minded professionals, you can go a long way closer to alleviating this weakness.
The more of us that display the bridger badge and talk about our commitment to the principals of briding, the sooner we will have clients entering the word Bridger in Google when they have a new project to begin or need help with an old one.

How do I do it?

There’s only a few steps and I will make it as easy as I can.

1. Choose one of the badge styles from below and then copy the code supplied beneath it.
Pasting this code into your web site or blog will produce the badge you chose with the links back to the bridger.

2. Fill in the form at the bottom and submit it to me for attention. Within a few days I will add your profile to my new associate page with a link back to your web site, blog or profile and I will include any keywords you provide to catch the attention of the search engines.

3. Soon afterwards I will begin to send you more members who would like to exchange links with you.

Note: If you have difficulties with the form below, you can simply give me your LinkedIn profile and I will create a profile out of this.

Option one

Committed to bridging the gap between business and IT Bridger

Option two

Committed bridging the gap between business and IT Bridger

Often operating as Business analysts or Project managers, Bridgers are committed a philosophy that says "1+1>2" and going the extra yard to speak simple English, Spanish, French, German . . more

Option three

Committed to bridging the gap between business and IT

Often operating as Business analysts or Project managers, Bridgers are committed a philosophy that says "1+1>2" and going the extra yard to speak simple English, Spanish, French, German . . more