How much data is “Big Data”

For many, this question is almost an irrelevance. The question that should always start the conversation is “ What do you want to achieve?”, yet in my personal experience it never has and when I have introduced it I have been made to feel uncomfortable. Many feel that they must have a big data project in their portfolio and the why? and how? is of less importance. A high proportion want answers to fairly simple questions they can’t currently get answered and are lead to believe that answering those questions is indeed big data.
Let me make this very simple. With few exceptions, there is only one reason why you might want a Big Data solution: Because you have so much data that anything less could not analyse it and provide solutions in the timescales you need.  There are two key elements here; Timescales and volume of data to be analysed.

Timescales is the simplest one so let’s deal with that first. There tends to be two timescales: 1. Instant response and 2. Non-urgent responses. The latter is by far the more common and is typified by the “Data warehouse” approach. The former is typified by the “search engine” scenario.
Although the search engine appears to be providing instant response, in reality it is merely searching well-ordered indexes that have been populated at a leisurely pace, so in fact it does not differ as much from the data warehouse scenario as one might at first think.
The data warehouse is a model of efficiency where the questions are carefully defined in advanced, most of the processing done and the answers stored away until needed.  Often further processing is then carried out at the point of consumption.
Again you may be thinking that there are more parallels than differences between the two approaches apart from all the hype. You’d be right.

What does Big data do that is different?

Well the term as we understand it, owes its existence to Google’s own solutions to the search engine problem. Perhaps another penny has dropped for you now.  Hadoop, Map-reduce and all those sexy terms refer to a simple and very powerful approach to getting a huge job done efficiently.

The infrastructure relies on the idea of dividing each job into smaller jobs and continuing to do so until each is quite manageable and then delegating them to different machines. If you’re a software engineer, think Jackson.  A simplified view might be that you have five people doing operational work and a manager coordinating that work and responding with a single answer to his sponsor. If you ever attended management courses you will surely remember this type of organisation. Well that’s the big idea.
Why is this better? Well it allows a vast, unlimited number of servers to work on bits of the problem at the same time,  thus speeding up the time to complete. This allows one to demand immediate answers to questions that are more efficiently dealt with over a longer time-frame and there lies the risk.
Is Big data Machine Learning?
No, it definitely is not, but of course it can be useful for doing this. However, it is very important to understand that there is a plethora of tools, many free and some you already have in the toolkit such as excel,  that can very effectively carry out machine learning tasks if you take a little time to learn them. Not only  is it  infinitely easier to learn and carry out such analysis on tools like SQL server, Excel, etc. than it is to spin up a big data factory on AWS and become a data scientist just to find out if it will rain tomorrow.

Very few questions you are likely to want answers to require anything more than traditional statistical approaches or even simpler BI reporting that can be carried out very effectively on stunning data volumes and extremely complex problem domains with tools like EXCEL (try SOLVER or explore the many regression functions), POWERBI, KNIME, RAPIDMINER, MATLAB, OCTAVE, Google FUSION TABLES, TABLEAU. Many are free and very good tutorials can be found online. The best thing about these tools is that you can test your hypothesis and decide whether a major project is worthwhile.

How big is big?

Well there are truly big problems and yours may well be one of them, but the vast majority of questions can be answered with a well specified windows server or your personal preference.
rememeber alos that remarkably small samples are known to provide extraordinaty insihts that improve very little when expanded.
For a better technical analysis than I could offer have a look at this very good blog. It gets to the point much faster than I do

As usual, comments only via email. No new subscribers being accepted at this time.

Let’s talk turkey or rather cloud

in
First me let me make it clear that I am a fan of what cloud and internet can do for businesses of all sizes and have been “bigging this up” for at least two decades. I would risk shouts of Hypocrite were I not to say so. Having said that I must warn against jumping blindly onto a bandwagon without even asking about the destination.

You may say I am being reactive and hasty, but I promise you that my concerns are raised by the sight of intelligent and knowledgeable men and women blindly buying into arrangements that are often far too risky at the current time in their current form.

I won’t name names or even suppliers, but be assured I have been dealing with global clients and global suppliers and this is what, more than anything else has rung my alarm bells.

  1. What if you fall out with this cloud supplier? (We all have seen or been involved in deals worth tens of millions or billions that have gone horribly wrong, let’s not live in denial) .
    What does the small print say about your rights in this situation? What if your supplier goes bankrupt, or simply withdraws from providing cloud services? Have you asked how you can get your data out in any case? How long would it take you to get back to business as usual in the event of any of these situations? How central to your core business is this cloud service? What else depends on it? Need I go on?
  1. The cloud only works when you have connectivity. What guarantees do you have of connectivity for all the hours when you will be paying your staff and hoping for productivity? And waiting for customers to call. Are you in denial again about the “connected world”. Last month I touched down in 4 countries and only once did my phone behave itself. Google maps, forget that! Calling a cab, not unless there’s a courtesy phone. Internet, nothing doing.  Boarding-card on your phone, I’d never have got started if they hadn’t used the good old paper passport to look me up on the system. Two weeks ago in Western Europe I was barely able to load a web page let alone be productive at work. I won’t go on, but lets, just say, you might one day be the only one who doesn’t have access to your data.
  2. Who sees your data? Everyone who wants it is the simple answer. First of all, the US government accesses any data anywhere whether on US servers or not with very few exceptions and I suggest you google how many major data breaches occurred with US government data in the past year to work out who else can see that data. Check out the stories of private and sensitive documents from drop box and google drive available to anyone on the web. There are two takeaways. 1. If someone is doing it regularly, anyone can do it, theft is rampant. If your data needs to be private and it is valuable to someone, keep it off the cloud. 2. If they can take data out they can probably put data in, that could be even more problematic.
  3. Who said the cloud was good value? Start-ups needing to get off the ground without the big capital investment love it, anyone needing to rapidly burst to meet demand peaks can find a great deal, however, many who have made this jump are feeling nervous about the level of costs, and the direction it will take. Certainly unless you scrutinise the details of every potential charge and penalty, you could live to regret it. I have personal experience of this. Yes there are tempting deals, but they are not all what they seem ad there is a big picture regardless of what the salesman says.
  4. Loss of skills, processes and people is never trivial. Unless you are very confident indeed about your commitment to cloud and more important the cloud’s commitment to you, then you really have to consider the risk of finding yourself without an IT department or even the memory of how to set one up as you find yourself pushed further and further into a corner by a maturing oligopoly of big players gaming the marketplace.
  5. Poor APIs and expensive difficult integration.
    Let’s be realistic, many businesses bought into the ERP idea thinking it would solve all their integration problems and they wouldn’t need an IT department. What really happened was the exact opposite. The demand remained or grew and the time and cost and especially expertise levels to meet these demands now quadrupled. Integrating ERP with anything is an area of pain I often wish I had never got involved with, but at least we had low level access all the way down to database level when necessary. A cloud ERP I can assure you, just in case you have not experienced it is an altogether less flexible beast than the original and fills none of the gaps but if anything exposes more. ERP was just and example, one way or another your business requires a lot of specialist systems in order to function and these needs will continue to change readily, but more importantly these systems will need to share data and communicate at real time. This capability is a long way off for the most advanced cloud platforms. They are effectively islands, closed shops. Some APIs are better than others but none I have seen are adequate and API’s alone are not enough to provide the level of integration needed.
    A few vendors are attempting to bundle middleware as part of the offering, but they are rightly nervous about making big promises. In fact this aspect of your emerging cloud infrastructure is an even bigger and more expensive challenge in many cases than in the old regime.
  1. Lack of flexibility plus arbitrary updates and changes.
    Just like your new Widows 10 install, you log in to do something only to find your favourite interface has vanished without warning and you can’t do it anymore. This is the joy of cloud computing. Would-be Oligopolies flexing their muscles and gaining in confidence in the knowledge that there is nothing you can do about it. Perhaps I jump the gun, but experience tells us that if they get away with this for a little while, and who will stop them? The next phase is likely to be complete  disregard for the customer. Speaking of which, have you ever wondered which customer they talk to before they design the next feature for “YOUR” business.  Wasn’t productivity software supposed to be about gaining advantage and differentiation …?
  1. IT without a medium and long term strategic view is a train-wreck about to happen, I doubt anyone with argue with this. Business stakeholders and IT people do sometimes disagree and there are times when business demands are not realistic or safe and other times without doubt when IT stakeholders are in the “Not invented here” camp, but as long as the dialogue continues, things will work out. The problem often introduced by the cloud model is that pre-sales go right past IT to talk to technical amateurs in the business arena  only engaging when they have already agreed a large investment and with no regard for the bigger picture. This can never work out and should never be supported by business at any level, nor should it be encouraged or even tolerated by Cloud vendors with a longer term strategy. Enterprise architecture is more important than ever when cloud solutions are involved.

 

Tomorrow, I could easily write you an article explaining why the cloud is a good idea as indeed it is, so please don’t assume I am saying that it is all bad, or suppliers are all bad, I am simply trying to impress on you the need to take reasonable precautions and think it through before jumping.

 

 

 

 

 

 

 

What to watch out for when researching and testing journeys

Previously:

Why you need to pay attention to customer experience

What to watch out for when researching and testing journeys

 

1. Last Click  can be a costly mistake

The customer journey does not begin with the last click before they visited your ecommerce site. No assumption could be more damaging than this one.

Here’s an example of what happens before “LastClick” (Uses App to order)

 UJ1

You need to take off that blindfold and clean your glasses if you’ve been ignoring this part of the journey.
Once you have got ahead, next you will need to criss cross this journey with the interactions with your competitors if you want to stay ahead for any length of time.  And don’t forget all those potential customers who have been influenced by this on-line saga.

For an interesting interactive tool that can be filtered by industry classification to get a glimpse of how last click relates to the other channels and influencers, check out the ink below

http://www.thinkwithgoogle.com/tools/customer-journey-to-online-purchase.html

 

2. Its not what they say, or what they do, or even what they say compared to what they do, but it’s what they tell their friends that is interesting.

If you have spent any time working with professional market research you probably noticed that people tell you what think you want to hear, what they think makes them sound clever, or mostly what they believe the group will be impressed with.

It takes a great deal of skill to capture useful insight from market research and to make large investments on the basis of just a market research report, however good, is a fatally risky thing to do.
People invariably do entirely different things to what they expected they would do in a particular situation and even from what they planned to do.  It can be very difficult to explain why you did this, you can read the underlying Social Psychology theories and decide whether to believe them, or not, but it wont change the facts at all.
People are more driven by their acceptance in, or leadership of groups than any other thing and hence, once they announce to the group what they are going to do,  it is more likely that they will do it than not.  Hence the most valuable insight into what someone will do is what they are wittering about in social media.
Be warned of course, just like other forms of research, this is one more voice to add to the conversation, not the single source of truth.
Just in case you are thinking it, most people ask this , there is a big difference between what people say in idle chatter to their friends and what they tell you in focus groups. The latter is not populated with their peers and it ends when they leave.

 

3.    The journey does not end at purchase, or anywhere close to it.

If you have been following the series so far, you will remember that in paragraph 5 of the last section I presented some hypothesis about the power of referral from your customers.  This is a very valid argument indeed for being switched on to the journey after purchase. In many sectors of course ,there is an ongoing relationship with the brand.

First of all the product may be one that is repeatedly used over time, each repeated use will therefore be another interaction with the brand and the quality of that interaction does count. Additionally there will be the difficult period when customers are learning to use your product and later there may be issues that require active support.

If you take the view that once they have purchased and you have their cash, there is little motivation to spend more money on them, then you are greatly mistaken, because even if they never plan to make another purchase of your product, they will talk to everyone they know and their influence will cost you dearly. Put together enough of these negative voices and it will put you out of business.

A far more constructive way of thinking is to ask yourself what up-sells, or cross-sells you might be able to introduce as a result of providing high quality personal support. If you don’t have other products, this may be the catalyst you needed to re-think your brand and product strategy

4.  Pay special attention to understanding Critical touch points (Moments of truth)

A user journey can be and usually is very complex, even when we only represent the main points. Even this however, is a potentially misleading picture. Thee are always Moments Of Truth when you win or lose on the tiniest of margins and these are the ones you must identify and focus your closest attention on.

For example: When the customer finally returns to your store feeling really happy with her decision, clicks buy and you don’t have her size in stock. That’s not a good situation.

When she clicks buy and gets a spinning thingy that goes on and on. Not good.

When she buys and although being a regular customer she is asked to enter a lot of information yet again despite being on a sunny bench with her HTC One and her longest nail extensions. You are really pushing your luck here.

It’s not all about clicking buy, it could be when she decides to see what people are saying about you and the first thing she finds is a blog by an irate customer of yours that you didn’t even bother to respond to let alone apologise for. The writing is on the wall.

These moments of truth may differ for different customer segments and personas, so you really need to know this and it may even differ at different times of day or seasons or with different devices.
For cheap flights customers I would say that, certainly for me it is “Paying the bill”,” Leaving on time”, “Arriving on time”.  The other stuff in this case is far less important and if it can only be done at the cost of these three, I hope they just ignore it.

“Learn what the Moments of truth are for your customers and focus a lot of your attention on getting them right, it will make a huge impact on your top and bottom lines.”

ZMOT-GRAPHIC1

According to Google, there’s a Zero Moment of  Truth.

“that moment when you grab your laptop, mobile phone or some other wired device and start learning about a product or service you’re thinking about trying, or buying.”

When I talked previously about taking a hit to let customers realise how badly they want your product, this was ZMOT.
When Henry Ford gave them a smelly noisy Iron horse and changed their lives, that too was ZMOT.  For you and I, winning at ZMOT means deeply understanding our customers habits and preferences so that we can be there at the moment when the inspirtaion hits them and showing the the right messages to take them to the next level in the journey.

1MOT

In bricks and mortar terminology, this is nothing more or less than “First Impressions”.  Until Google gives you the time machine, you can’t go back and redo first impressions.
We spoke in the last instalment about how People see what they expect to see and feel what they expect to feel,  well this is ground zero. Here is where you set that expectation and it had better be good.

2MOT

This is when they open that box and your shiny product falls on their toe and puts them in hospital, or they struggle with one point white print instructions on grey papaer in broken Korean. You may well have thier money in the bank right now, but dont fool yourself about the importance of getting this right.

3MOT

Is the remebered sensations of using your product and how they compare to the expectation.  Wow that requires some thought.
Its not enough to have a great product, but it should be at least up to the expectation. Not only do they need to like the product, but they need to trust you to give them an even better one when your competitor moves the goal posts with a new offering, otherwise you’ll be saying an early farewell. Don’t forget that it is not just your product, but everything your business does in the public eye as well as how you answer your phones, or not and help them when they have a problem.

4MOT

This is when they talk about you to their friends and social media contacts. This is one you absolutely must win

Integrating a large complex enterprise without duct tape

Enterprise Architecture Integration

EAI is not just for technical people, it affects everyone.

This article is aimed at senior executives with enough interest in the subject of Enterprise Architecture Integration (EAI) and sufficient grasp of the very basics to be able to gain a workable understanding of why things are the way they are and how they can be fixed.

It’s not an article for IT pros, though occasionally a little extra is added to accommodate those who may be lurking, but ignoring the odd technical passage should not prevent you gaining a useful insight . After all if you decided to read this you are either in deeper trouble than you know, or you do have some understanding,

So what is wrong with how we do things now?

I am answering this on three levels to help those who want clarity and to accommodate those in a hurry who know what they need.

At a strategic level

Strategy development canvas

 

 

 

 

 

 

 

 

 

 

Information Technology has hardly been around long enough to be strategic. By its nature it attracts three types:

  1. Those who pretend to be followers, but really want someone else to handle it, like we might farm out the legal stuff. These are Hiders.
  2. Those who want to play with it and are not concerned with what it actually delivers in terms of benefit. The Players
  3. A few souls who are determined to make it pay by focusing on benefits and understanding the potential. These are the Entrepreneurs

Probably 80% of all the c levels I have met fall into the category of hiders. A nice silo in the next building with the very latest ticket system and that’s that covered.  Actually no, that is ignoring a core part of your business and you can see why if you read on to the next section.
At least 90% of all IT directors and CIOs I ever met were “Players”. There’s nothing wrong with enjoying your work, in fact it’s important, but we go to work to deliver value as part of an overall strategy and plan not to amuse ourselves, so the onus is on the CEOs to drive this agenda and engage his/her key people to the strategy.

Business strategy as a driver for architecture strategy

Before engaging in Enterprise architecture, it is critical to be in possession of the very latest business strategy and if such is not available, or it is somewhat stale, then you must start out by engaging leaders in a process of thrashing out high level strategy sufficient to inform you architectural designs over the one to five and the five to ten year period s.
Remember, strategy is not a detailed plan, it is by its nature high level and is easier than you may think once you get people around a table .
Exercises like SWOT and PEST are a useful tool to surface key inputs.  Focus on constraint based modelling and usually the space in which you are free to operate successfully dictates the strategy options you have.

At a process level

Software solutions are tools that make processes more efficient and more accountable. Processes are in the main tool by which every business creates value and when they are intelligently optimised and as much as possible automated, they create a lot of extra value.

At all other times Software solutions are a barrier preventing change, soaking up money and effort and worst of all creating illusions that are damaging. People “working the system” instead of delivering .  e.g. the person too busy fiddling with the CRM to speak to a customer.

If you are not interested in the process your business adapts and follows you are not in charge and you need to re-evaluate,

political integration

At a technology level

First we had a system- eureka, then we had two systems and they didn’t talk to each other.
At first we thought we could develop our own systems around a single database and all would be well since they all used the same database. A few SMEs managed this at huge expense before admitting defeat mainly to the cost of bespoke development.

Big software vendors then saw an opportunity to tie us all in and they went on a trail that has ended up in ERP ( all things to every business) and because they are all in one piece, they do talk to each other internally, but woe  betide he who wants their ERP to talk to something else. That’s was never the plan, was it? You can have a SAP business or an Oracle business just like your competitor, but you dare not try to differentiate.

Then of course modern businesses like to make acquisitions and disposals and this drives a need to be able to fairly quickly connect an external business or cut it loose again as market forces dictate.
Increasingly the success of outsourcing process relies heavily on integrating process at just the right level and doing it fairly quickly, securely and without prohibitive expense.

Technology silos create process and culture silos and the one that usually feels this most sharply is the customer. Silos are never ideal, but they can be accommodated in a federated business architecture and integrated more closely over time, this latter is very important to an acquisitive business.

Technology must be driven by principals that focus on creating value no w and in the future, it must deal with the now well while keeping one eye firmly on the future.  It should be driven by the whole management team as a key strategic asset with a workable interface between technical specialists and business process specialists. Nothing less will deliver.

The problem with this is that it leaves the strategist nowhere to hide. If the strategy is to go east and the rocket is built then it can’t simply be turned around to face the other way. This is not a weakness in technology but a fact of life.  Building a new office block or fitting out a manufacturing plant is no less prone to the impacts of change, but nobody expects to pick up their building and move it a few blocks while rotating it.

Businesses that do well with technology have very strong strategic direction and operate in mature markets where a large portion of their activity is stable and fairly slow to change, or their business is software.  The best ones have ability to shield critical line of business technology assets from a more innovative and risky portion that is contained and risk managed. All businesses that win at technology have realistic expectations and an acceptance that technology is a huge core part of their business, leading to more fruitful relationships with technology experts and suppliers.

Apart from strategic planning and attention to relationships with IT, there is an awful lot that can be achieved by IT through investment in Enterprise architecture capability.
Engaging architects to own and drive the overall vision while maintaining the maximum flexibility to integrate new systems and replace underperforming systems is a key strategy in any IT estate.

 

 

What options do we have?

 

Strategic Options

Planning
Strategically we must not get side tracked by people claiming that workld has gone agile and it is OK to do everything last minute. Strategic planning is just as important as it always was and the less change we drive through about turns and poor planning the better our results will be.

 

Communicating

As in all things, communication is the lubricant the glue and sometimes evens the fuel. Unless you develop the capability to analyse and define problems accurately and to evolve solutions innovatively then all the technology in the world will create nothing but cost.
Business architecture is the domain of Enterprise architects and Business analysts. The work they do makes sure that the processes and teams you put in place are a very good solution to the problem and that when the systems are released to automate and support these processes, you get maximum value from your investment.  If you are in the world where vendors set your strategy and Geeks select your systems then you probably view systems delivery as purchasing systems and implementing them. This is the lowest level of maturity and you need to seek help with moving your capability along as fast as you are able to contend with.

Architectural Options

Segregating

Segregating our architecture according to the propensity for change gives us a chance to maintain a large portion of our architecture in a very stable state with little change affecting it, maintain a layer of stable but more changeable architecture and finally a small layer of highly agile architecture that is designed to be charged rapidly without negative impacts on the remainder. This latter is the key.
More than 60% of IT spend goes on dealing with the impacts of relatively small changes on a monolithic architecture.

Below is a representation of a well evolved architecture principal that enjoys security and stability while allowing agility and innovation in a safe environment.

Typical industry architecture might include CRM and ERP systems that shoulder the brunt of the heavy lifting while the Organisational layers can include focused solutions for specific problems . Mashups developed from REST and SOAP APIs is a great example of organisational architecture that is fast and cheap to develop and creates the added value without introducing unnecessary risk

Architecture continuum

 

 

 

 

 

 

 

Only through maintaining an architecture capability can you hope to achieve this level of robustness and freedom.

Separate systems can be integrated in three ways potentially:

User Interface level

Portals and Mashups
Imagine the unfortunate  person and there  are many of the, whose work forces them to log in and out of many systems continuously all day with a pen and ad ready to grab ids and reference numbers before searching the next system and gradually solving the puzzle. I have sat many times with people such as this and I have utmost sympathy. Sometimes they say,” just get me a single log-in point and you will change my life”.

Sometimes a simple portal where links to all the systems are in a single portal page that remembers credentials and logs them in automatically is a stellar staring point when there is not the will or the finance to do better than this. A clever web developer can sometimes achieve this for a modest investment and it does deliver value.

One step further is to access APIs and scrape web pages or even windows screens in the Background to get at the data you need and present the user a single interface that secretly deals with all the others in the background.  This is more complex and more difficult to maintain, but sometimes it can fill a gap and can also be valuable in proving the value proposition before investing something more stable

API level

Most major commercial systems built since the mid 90s have an API of some sort using COM,  CORBA,Web services, or some other form of communication to offer access to basic processes in a safe and reasonably painless way. Some more recent systems provide fairly complex web service interfaces offering  REST/SOAP interfaces.  Of course there is still work to be done to get around firewalls, maintain security and of course write and maintain the access code and as things change previously implemented code must be moved and changed.

The advantage of the API route is that the business logic protecting a user interface will normally be also present behind the public interface and therefore protecting the integrity of the system from broken updates and inserts etc,   but on the downside, these connections to APIs are by nature, but not by necessity, synchronous calls that gradually que up a time bomb of delays that will eventually sink your architecture and bring it gradually to a standstill.
What do I mean by this?

Well, each time a system calls another, unless in special circumstances, the caller must wait for the callee to respond before that thread can do any other work. Typically an Operating system will have attributed memory and a process thread to the caller and these resources are now out of commission waiting for the callee to respond to the callor. What if the callee is waiting likewise for another system to respond. Not only is this a sure way to cripple your architecture but is nigh impossible to track down and fix once it has been allowed to  get out of control.

middleware

 

 

 

 

 

 

 

Database level

When all else fails, you have the option of integrating at database level. What this means is that:

  1. You must identify precisely the data you will need from a system, then work out exactly how to access it and write a query to get the data and present it to your target system.
  2. Transform some or all of the data so that it will fit in the target system
  3. Write a query that places the data into the Target system in such a way that it is usable and it does not in any way damage the integrity of the target system.
  4. Often you need to execute these two queries as a single transaction with ability to roll back if an error occurs
  5. Test all of that and ensure everybody that there won’t be extra zeros on any accounts or thousands of lost customers or the any other horror stories.

The first step is often enough to break the strongest will. E.g. Oracle databases behind an ERP is an example of a database with no semantic naming conventions and requires you to look up the table name in order to know which number table combined which other numbered tables holds the data you need.

The final step will be like the first in reverse only this time you have to work out a bomb proof test strategy so you don’t discover the error after you have leaked millions of pounds.

Modern databases are relational meaning that data is normalised in very atomic classifications that drive the table schema. To add a small piece of data often requires updating several tables in a particular order so as not to create orphaned data and break the integrity of the target database.

The API method we discussed previously has already handled all this complexity for you so you can surely see the attraction. The problem is that APIs almost never provide sufficient access and often they simply don’t exist.

 

The nitty gritty of integrating systems safely

Now that I have got your feet wet, let’s have a look at the problem from a logical and conceptual viewpoint.
Let’s say we have 20 separate large systems ( a smallish enterprise) that overlap considerably in their business capability and are used by different departments, or even companies in the group. It is reasonable to assume that all are closed systems with proprietary databases and about 50% of our integration needs can be served by the existing APIs, but the rest we have to solve with Various connectors/endpoints.

It is reasonable to assume that being closed systems they will use different semantics and different data structures, depending on the database type behind them and the development language the apparently similar data types can have different sixes and characteristics.
e.g. We have an SAP erp system that stores FirstName, Initial, Lastname  and a CRM that stores name as a single string with a space in the middle.
We now have different names for things and different structures as well. This will be repeated many times over between the 20 systems.

Let’s suppose we need all 20 systems swapping customer and product information.
The number of point to point connections would be n x n-1 or 380 connections.
Each system would need the ability to translate between its own names and structures and 19 other vocabularies.  That’s an awful lot of programming.
In the architecture world, this is what we call the hairball effect.  Take that 380 connections and 19 vocabularies and apply Moore’s law and you are nowhere near grasping the extent of the problem but it should be sufficient to open your eyes.

To make this whole endeavour manageable, we must solve the biggest problems first and reuse these solutions.
Component s of an integration project

  1. Translation between vocabularies can still cripple teams, let alone systems, so we must deal with this one first.
  2. Channels need to translate between transport protocols too

messaging pattern

 

 

 

 

 

 

 

 

 

 

  1. Stacked up synchronous calls between systems each of which is waiting for the previous is crippling and invariably the circle gets completed resulting in melt down. We need an elegant reusable solution to this problem. Publish and subscribe is one example of an integration pattern that solves this problem while being easy on resources.

 

pubsub

  1. With the core infrastructure in place to tackle integration we can then seek to reuse connectors from the marketplace for the better known systems and thus reduce not only the effort, and duration, but the levels of risk.

Translation

Translation costs can be reduced dramatically by using a Canonical model as a midway step.
Instead of needing to understand 19 languages, each of our systems would only need to translate to and from the canonical language.
That reduces the previous estimate of 380 to just 40 translations .  Not only is it a huge saving, but it avoids the folly of trying to coerce other systems and teams to a shared vocabulary.

canonical data model

 

 

 

 

 

 

 

 

Transport

What we need is a simple protocol friendly transport system that accepts any protocol in and delivers any protocol out so that when combined with transaltors,all systems in the architecture can converse freely.

multicast

 

 

 

 

 

 

Distribution
Getting messages to systems at or before the time they are needed is the key to good integration.
Think of an email server for a moment. I can send my messages any time that is convenient and my colleagues can collect and read those messages when they need to. When we relied on telephones, we had to be at our desk when it rang or we missed the boat. Email is an asynchronous method of delivering messages. It has Queues where your messages can wait for collection or wait to be sent and it can be configured to keep sending until the message is received and read. Guaranteed delivery ( I have in fact used an email server to integrate ships at sea in just this way).
To make a good integration system we need just a few more bells and whistles to help poorly equipped systems access their messages or in some cases to push the message right into their stores. Sometimes all we need do is to inform all and sundry that a new order has arrived and any system that needs the data can come and get it. This is called multicasting and it is very efficient.

Putting it all together

Enterprise Service Bus ESB

The Enterprise Service Bus is a design pattern that has been latched onto by vendors offering a custom collection of tools to solve known integration problems. A commercial ESB will normally contain all the components you need and many optional extras. It should also include or make available connectors for the major popular systems out there such as ERP and SRM systems and many others.
The latest breed of EAI solution is the cloud based ESB. The benefit  is  that no maintenance people need be retained for this very specialised field and if a true cloud solution, there should be unlimited and effortless scaling available. Although annual costs can appear high, most vendors will negotiate and TCO calculations should bear in mind the cost of maintaining a n on premise solution.
It is probably worth checking whether you are being offered a true cloud solution or a virtualised  on premise solution  on the cloud.

Steps to designing the optimum integrated enterprise architecture

  1. Understand the problem at a strategic Enterprise level and defining the value proposition.
    Similar to requirements gathering in a solutions scenario, we must collate the problems to be solved and define them with sufficient accuracy to feed into accurate solution architecture.
  2. Addressing the viewpoints of stakeholders, resolving them with strategic concerns
    previously raised and identifying the concerns to be addressed and the ideal priority.

    The key difference from requirements engineering is that each concern or viewpoint must be handled individually and a business case made at a high  level so as to avoid the investment of a lot of time in problems that are not worth solving or simply too tricky. EAI stakeholders tend to be 90% strategic and 10% end user in a perfect inverse of solution requirements.

  3. Understanding and mapping the processes to be supported and improved.
    Until you have processes mapped, you simply can’t decide what data will be needed at what point and any integration you design without the guidance of process is likely to be hit and miss at best.
  4. Identifying the business events that kick off the processes you are supporting.
    Business events are a terminology used a lot in business analysis also. They are in effect the thigs that happen in a business that drives the need for a particular process to begin. Understanding events and how they are linked together is critical to understanding the strains on a system and the order in which tings occur. Without this appreciation your timing can be hit and miss.
  5. Identifying the data needed to support the processes.
    Out of events and processes comes the first steps in identifying the actual messages that will be needed and the metadata that these messages will need in order to be transported and processed. E.g. a new order in a store may be the vent that drives a purchase process, which in turn creates a customer message . The customer message needs certain metadata in order to be allowed past the security and to be linked to the account and product data in the target system. There are few shortcuts other than following the events and processes and analysing the messages required
  6. . Understanding where a data entity is created and through hat states it is passed and processed helps one to understand the best source of data at any time and to put best class governance in place. Defining a canonical meta-model to support translations between the many stores and systems is vital to contain the number of translators required.
  7. Identifying the performance required of the many integration flows and the volumes of messaging.
    This requires rough estimation of the likely usage and sanity checking against the available resources. For safety it should then be load tested to discover at what point performance is reduced to a floor level and plans put in place for maintenance and review
  8. Defining a conceptual model of the architecture to support selection, procurement and technical architecture of the solution.
    Identifying the key EAI patterns needed to support the architecture helps the architect to select the right solutions and avoid expensive bloatware
  9. Selecting the technology solutions and procuring them.
    Preparing the procurement documentation, Inviting vendors to participate, carefully interrogating them and spotting the weaknesses and in advance is a key aspect of getting it right.
  10. Planning and risk managing the implementation.
    Planning the implementation whether carried out entirely by the vendor by your own team or a combination is always tricky . Identifying with great accuracy who is accountable for what and having clear acceptance criteria and reliable testing and QAS procedures is absolutely key to success.
  11. Producing a test strategy that is fit for purpose.

A detailed test strategy should cover all the likely test cases with special emphasis on the intended use and immediate usage cases and act as a certification that the architecture has met expectations both functional and non-functional.

  1. Defining the ongoing maintenance procedures, capabilities, OLAs, SLASs , roles and responsibilities for the infrastructure and data governance concerns .
    With the results of testing at hand a review and maintenance plan should be put in place to ensure that performance and security are maintained at a high level and contractual agreements are met.
  2. Creating a benefit realisation plan combining business change and benefit measurement strategies to capture the benefits and to measure and understand them.

Returning to the mini business cases for significant integrations and to the case for the infrastructure, it is now imperative to establish KPIs that will reasonably be used to easier and report on how well the project has performed as an investment. Not every business case must be proven in financial terms but even the most non tangible of benefits can be measured where there is a will.

 

 

The DNA of CHAOS. Why software projects, peace negotiations and football games all come up short on occasion and the true value proposition of agile

Why Business analysts, Product Owners, negotiators, political reformers, scientists and many others are doomed to repeat the same mistakes until they drown in in their failures.

  1. Most people in a work situation, and indeed in most situations, live in a consciousness that is deliberately falsified to reflect (a) what they believe is expected of them by peers, and (b) what the asker of a given question wants to hear. This is a conditioned response and nobody’s fault.  Terms like “Group Think” attempt to explain it, but none really explain the phenomenon well and the long version is just beyond our scope in this piece. Market researchers are trained to try and avoid it, Politicians and advertisers learn to harness it. If you have the role of finding out what  a business wants to do so you can get the software designed to support, it, you will be faced with the same  aggravated problem. Asking and listening simply won’t get the job done.
  2. For reasons I don’t claim to understand fully, most people have a favourite solution in their minds for most problems they are aware of. Expensive  housing – they may say; “Stop immigrants coming in”.  The call centres is too expensive and eating our profits- they may say; “ Close the call centre and let them use Twitter”. Always there is an obvious political, or financial gain for the responder from this solution. If it is a senior executive, or politician, there will be cleverly argued points and tables of figures that make her pet solution look like a no-brainer.
  3. Michael D Cohen  coined the phrase “Garbage Can Theory” to describe this phenomenon which permeates politics, business, the playground and just about every area of human life crippling every endeavour to improve our lot century after century. If you are trying to get your proposal through programme and investment boards, or trying to agree a solution closer the coal-face you will drown unless you understand this phenomenon and take steps to work with it.
  4. Many researchers and writers have agreed strongly that the human brain makes virtually all decisions in a matter of time between milliseconds and 1 to 3 seconds, but then spend enormous amounts of time and effort researching in order to prove their decision was correct. Most people are entirely unaware that they do so.  This of course is not news to Scientific researchers who often publish the theory before beginning on the research.  In the academic world where we have to start somewhere this can be a useful approach. Product managers and entrepreneurs also start with the theory and are difficult to budge even when wrong, yet these leaders are necessary to get something started at all.
  5. People are conditioned to follow a strong leader and will convince themselves of their support for any theory put forward by this leader. This is largely an extension of  point one, but worth being aware of all the same when venturing into any group to arrive at a shared view.

These Five forces combined thwart just about every effort to form a good business strategy with technology in mind, agree a solution and still be in agreement when the solution is completed to specification even a few months later, let alone after a year or more. Waterfall and other methods of managing the software delivery process all rely on a simplistic view that if you gather enough requirements from users and stakeholders, you can then haggle over priority and build the killer solution to solve everyone’s problems. In reality, there is rarely any agreement, there is even less concern over business outcomes and near- total focus on the many garbage can solutions. Nothing this project could deliver is ever likely to meet the satisfaction of more than a small proportion of stakeholders, the others are always going to be loud in their dissatisfaction and only by sheer luck is their ever going to be serious business benefit.  This project, however well the techies perform will be added to the list of “failures” and the cause will be placed at the CIO’s door. Failure to focus on goals Developers and technical architects are driven by developing exciting new toys. That is their package in the garbage can. Techies ultimately want and are entitled to, clear definitions of what they must build.  They then build and test precisely what you asked for and give it to you working  very well. Few people would try to argue with that, let alone succeed.  The problem is that what you asked for does not always address the problem you needed solved and in fairness the developer or indeed the CIO is not to blame. There is one technique that all accomplished negotiators agree on whether brokering peace in a war zone or negotiating a business contract. YOU MUST KEEP THEM FOCUSED ON GOALS NOT SOLUTIONS! If you don’t, the garbage can will take over.
One cause  of the problem is not using the effective solutions we already have,
The need to focus on goals rather than solutions is not  new to the software industry by the way.  MSP, Prince2 and TOGAF are frameworks I am personally very familiar with, all of which, when used correctly, focus on outcomes required, the solutions that will best deliver those outcomes and the details of each solution, in that order. BABOK and the ISEB framework for business analysis also follow these same principals and are very clear about it. The problem is that few people make any attempt to use their training effectively, the training is rushed and focused only on answering multiple choice questions to gain a certificate and of course the majority of people involved in the business have no training at all. This same scenario, by the way, applies equally to Scrum. Scrum solves some of the problem Scrum attempts to alleviate the problem in a different way and with a similar level of success. Scrum is invented and owned by the technical community as a direct response to these issues plus the difficulty of communicating the complexity of technology with business people. The Scrum team wants only to be given an unambiguous requirement so they can do what they studied for, build great software. They place the onus on the business to appoint, or work with a Product Owner who will be seconded to their team and take all responsibility for the definition of the Product/System.   Communication issues are solved by very short Sprints of just two weeks typically and using prototypes and working software to describe things to the product owner in place of documents. Initially this arrangement solves the problems for the Scrum team, there is no ambiguity about what they must build, they do as much work as the Product Owner wants and stop when he is satisfied. There is no room for dissatisfaction with the IT team. Likewise for the Product Owner, if he is unsure, he can build something , test it on real world scenarios and then make adjustments until it delivers the right outcomes or scrap it altogether.
Inspect and adapt solves problems with lack of knowledge, but also masks others Scrum tries to tackle the strategic questions by replacing strategic decisions and documents with Inspect and adapt. For online products and for certain internal systems, this works extremely well because you can build the minimum, get it working and in use, learn from success or failure and add a little more, remove some, or start again. The risks are lower, failures are smaller and less expensive and the problems with groupthink, daydreaming and garbage cans are alleviated by an early sobering splash of cold reality that comes with release of a Minimal Viable Product. What is often wrong with this approach and fatally dangerous, is the lack of a strategic layer to the thinking process. Inspect and adapt is a close cousin of the OODA loop as defined by Colonel John Boyd. The principal is very simple that in a dynamic environment, he who has the most clear view of the situation always wins given a reasonable ability to make decisions.  Pilots flying inferior jets were seen to consistently win dogfights  because they had a better view from the cockpit and could make better decisions faster.  Developing OODA  (Observe, Orientate, Decide, Act ) and teaching it to pilots made significant improvements over giving them a detailed instruction, or a faster Jet because as the military have always recognised, in the field, the operative has superior knowledge in the heat of battle and must have the right level of autonomy and confidence to decide and act. No military  has ever cut loose a person, or team, or unmanned drone to inspect and adapt and just stay out there. There has to be a clear goal, a Definition of Done, Principals that guide it in the field and a strategy for completing the mission. Then Inspect and Adapt comes into play and in the same way that the human brain itself learns,  the actions  in the field can feed all the way back to sometimes cause a change in strategy or even a revision in Principals, but  actors in the field DO NOT  GO AWOL. Principals and strategy come first and Inspect and Adapt is a response to exceptions , it is not the  plan.

The real reason why Agile/Scrum is winning friends in the world of software and gaining attention beyond these confines

I say Agile/Scrum because when most people speak of agile they are really referring  to Scrum. However Scrum has encompassed much from other Agile implementations and definitely does not hold the sole franchise on Agile methods. The first problem Scrum had to address was the great chasm between Business leaders and Technology leaders.  Various versions of the same solution see the business seconding someone to the Scrum team, or colocation of business and technical people working on the same project. This is a perennial problem sometimes referred to as silos and secondment or colocation of cross discipline teams is a useful technique that helps, but doesn’t alone solve the problems. In the case of Scrum it forces the business into making decisions for better or for worse and living or dying by them. That helps techies to get on with building stuff, but it doesn’t, of itself, help the business much with getting the strategy right. I.E they produce more stuff, but don’t necessarily solve more problems. The next problem Scrum had to address is helping the business in dealing with the unknown. The speed of change is much greater in the past few decades driven by the fast pace of disruptive technology. Organisations are faced with problems they can’t analyse easily and therefore can’t solve. They also find that when they know the problem, they don’t have an obvious solution. This is the situation Scrum was designed for in its entirety. A Scrum team can make a best guess and very quickly get a potential solution in front of customers, Inspect and Adapt and return quickly with an improved effort until the right solution is reached by trial and error and then it can be built upon until it reaches its optimum scale. The biggest problem addressed by Scrum and that one that is causing all the excitement is the almost  total  lack of leadership throughout modern businesses (the past 10 years). Few would lament the passing of the great Command and Control organisations, but with them went whatever semblance of leadership we had in organisations. In the Flat organisation leadership in name is often cited, but leadership in deed is sometimes frowned upon or even punished. In the 80s and 90s business leaders bullishly wrote books about leadership and motivation. These were charismatic people with an instinct for what made others tick.   Kenneth Blanchard, Peter  Drucker, Stephen R Covey and their like gave us the ideas and the passion to change things and build things. We somehow lost this passion and insight in the transition.  Scrum is a beacon in the dark, because it values people and their interactions over trails of evidence. It values achievement over promises and it values self-improvement  and job satisfaction. Here and there Scrum teams are springing up that capture attention by their high productivity and ingenuity and win the envy of their colleagues for their enthusiasm and Joie de vivre.  People who see a good Scrum team in action want to be part of it or learn how to make that happen in their business.  That is the real reason why the buzz of excitement continues and why Scrum experts are being invited into other areas of business.

How to implement Scrum while keeping your sanity and your Job

Cost overruns, missed schedules and quality issues. These were the reasons cited for the development of Scrum.
Management also wants to harvest the business value expected from new software sooner, while using fewer resources

Breaking a traditional process up into small increments solves all of these problems. MSFis a wonderful example of this apart from the fewer resources bit, that is a myth. MSF encourages the business to develop a long term vision for the solution, but then it is broken into a series of small deliverables that build up in timeboxes to a full solution, or stop when time or budget run out.

The benefits are significant because; a. A proper Enterprise Architecture, style and vision is agreed first. That means that we don’t start off with one approach then half way through have to scrap it all and start again in order to scale. B. It means that at the end of the first timebox there is something of business value to show to the customer.

MSF differs from scrum however, in that it still depends on some basic documentation and designs and formal testing in a way that is familiar to most software engineers. MSF also lends itself well to procuring systems which I can’t honestly say about Scrum.

The downside is that MSF does not greatly improve communication other than the fact that it keeps the timeboxes short and encourages a collaborative agile team of peers, but the documentation still poses a threat to communication.

Scrum is a very different kettle of fish.
Scrum is designed to address a number of other problems:

  1. The Product Owner is a response to the lack of communication in traditional projects caused by remote stakeholders and by the sheer challenge of communicating a business idea from an executive to a programmer or architect accurately. Putting the business person in the role of Product Owner broke through this barrier.
  2. Demotivated and remote “techies” interested only in playing with gadgets is also a perception that still exists and it is countered by empowering them, making them into generalists and putting them in daily communication with their customer in the business.
  3. Fear of not knowing what “they are doing in there” is still an irrational fear that plagues the relationship between technology and the business. Delivering demonstrable pieces of functionality at regular intervals lets the business see proof that they are working and usually sends them off to sleep within a few months.

Aspects of Scrum that pose serious threats to success of a project

  1. Scrum was designed for writing code for fairly small projects. It was never designed to be used for many of the other unbelievable uses it is put to. The reason for some of these ridiculous applications of Scrum may be that Project management skills have been deprecated in favour of” the new thing”. The chances are they were not great to begin with and hence the adaption of Scrum
  2. Scrum is NOT Project Management, it’s designed by software engineers for software development. In small projects, the Product Owner could carry in his head all the stuff normally entrusted to a project and the scrum master can run the development, but this does not translate to bigger projects.
  3. Scope creep. Yes, we thought we had beat this one but in fact the reality is often that Scrum introduces a crippling level of scope creep. Three reasons mainly are to blame:
    1. No formal change control, nobody to say ” you are kidding?” so everything goes on the backlog
    2. No formal end date, so as long as it is all cosy, the business customer will keep on going, why should he stop?
    3. Bugs and millions more of them. Scrum teams don’t do formal testing and if you ever developed software you know what this means . Scrum teams also decide which bugs to fix and which to ignore. Naturally this becomes a tool when meeting the end of sprint deadline, it may be that the system works, but only just. That is acceptable by the way.
  4. A Scrum of Scrums is like replacing a big house with a tower block. It isn’t smaller or more manageable, it’s a mess. We spent the last twenty years turning old fashioned command and control organisations into flat collaborative ones. Now the scalable Scrum has built them back up again with a cell of cells like a medieval army. I’m not saying its impossible, just that it needs improvement and it’s not always worth the effort.
  5. Estimating is very difficult and Sprints are far worse at predicting their time and scope than traditional projects ever were. Costs are sometimes spiralling rather than falling, they just are not visibly so.
  6. Frequently the techies dominate the Product owner to get their own way and the collaboration is little more than a nice theory, the difference is he won’t tell anyone.
  7. Scrum was designed to be done by very experienced engineers all of whom were trained in Scrum. In reality few team members are even certified and the engineering skills are generally not good enough for a self-contained empowered team
  8. If any of the team members leave it can have a serious effect on the outcome.
  9. Scrumbut is a major problem. It refers to the practice of leaving out key parts of the process because you don’t like them, or whatever excuse. The problem is that velocity charts, retrospect and all the other rituals are there for a very good reason and when left out you end up with a traditional bunch of techies running amok with no controls, planning, or performance improvement strategy.
  10. The hatred of documents and authority figures really hurts when it comes to architecture.
    The team gradually becomes braver and makes more unqualified assumptions about the underlying infrastructure they are building on until eventually disaster happens and when it does there is no documentation, the designer has left and there is chaos.

 

So what should do you need to do if you are considering a Scrum implementation in your organisation?

  1. Begin by spending as much time as necessary to be very clear about what problem you hope to resolve with Scrum, or what benefit you want to reap.
    Don’t implement Scrum to deliver major projects, it was never designed for this. If you want to add some of the elements of incremental delivery etc, go ahead and introduce them to your traditional approach, or explore another method such as MSF.
  2. Define your criteria for deciding whether a project should be approached in a traditional way or via Scrum. Get advice if needed and especially in the first year, be very careful to tackle only projects that lend themselves to Scrum. As a guideline I would say:
    1. They should have a small element of invisible work, i.e. if there’s two months work to implement stuff you can’t demonstrate, don’t tackle this one with Scrum at least not till you are very experienced.
    2. The solution must be something that lends itself to incremental delivery and can be meaningfully demonstrated.
    3. The whole team including Product Owner can be located in the same building and I would try to get the Product Owner relocated to sit with the team.
  3. Define a clear framework for Enterprise Architecture and let the Scrum team clearly understand what they can touch and what they can not and don’t budge on change control mechanisms when it comes to mission critical systems and those that impact them.
    In TOGAF terms a good approach might be to say that only the Organisation specific technology can be touched and within that I might label parts of it uniquely for agile access.
  4. Don’t start with a team who have let you down at traditional approaches and read an ebook on Scrum. Send them away for a few days to be trained and hire an experienced person to coach the team through at least their first project.
  5. Put in place a QA process to monitor Scrum and their performance. Make sure that all the rituals are being done and in a meaningful way. Make sure the velocity is being monitored, monitor the accuracy of estimations, watch out for debt in the bugs department.
  6. Most of all, when you have a good plan in place and you are satisfied that you have the answers to hand, present your plan to your colleagues in the business in the context of how it will be impacting them, how they will benefit and how they will be expected to change the way they engage. Make necessary changes and then plan for the launch.
  7. This is really important. Take a cold look at your existing project management processes and ask yourself what is it that made you so keen to try an alternative approach. If some of these issues are not going to be solved by Scrum and I predict that will be the case, then set about resolving them in another way.
  8. Make sure you have a clear interface defined for how Scrum teams will interact with traditional projects and Programmes when asked to do so. There must be no them and us, but on the contrary, the two disciplines must support each other. You will need your programme and project management capabilities for many things and it will be beneficial to be able to interface a Scrum project with part of a project or a programme without ruffling feathers. This will also save you from the ignominy of trying to deliver huge projects via Scrum

 

TOGAF – well I finally did it ( here’s the summary)

TOGAF
I couldn’t find a readable short summary anywhere so I decided to bear the pain, skim read the dull text and write a summary for any unfortunate person trying to get a quick overview of a new framework in an industry he/she already knows well.
My conclusion after a few hours of wading through the treacle is that it is well put together, if horrendously poor in it’s presentation.
A slight contradiction is that parts are already a bit outdated in that it pays too much attention to things like portability in a world where interoperability is the buzz word and the “cloud” and “tower of babel” are the modern metaphors, nevertheless any slight shortcomings can easily be overcome by intelligent people using it.
A sharp warning though is that; even more so than the ubiquitous Prince2 for project management, TOGAF won’t make an architect of you, you have to bring all the skills to the table and then it helps you get things in the right order, but even that is up for interpretation, so you really must know your stuff and have integrity
It is not therefore a reliable hiring tool in it’s own right and I see it as definitely a team game.
TOGAF calls itself a Framework, but the main body of it is ADM which is in fact a process for doing architecture.
It follows the usual pattern of breaking up enterprise architecture into four logical steps (steps because it is a process)
The technical architecture uses a method closely allied to the idea of inheritance in OOD whereby the patterns and principals are inherited from more generic ones and you then build something more specialised on top of all that.
E.G.
Organisation specific ( This is the innovative bit)
Industry specific ( This is common to all organisations in a specific industry and saves a lot of time)
Common Architecture ( This varies little from one organisation to another therefore requires a little more effort than foundation, but saves much effort)
Foundation architecture ( This doesn’t vary much from one organisation to another therefore requires little, or no effort)
They call this idea the “Enterprise continuum”
Questionable ideas
Things I would question:
1.The Foundation architecture specifies knowledge bases that you need to develop such as a set of standards and a Technical reference Model. This part is potentially futile in that:
a.constant change often makes last years standards out of date, but nobody ever updates these things
b.notwithstanding the previous comment, experience suggests that few people ever refer to them even in some big name software houses.
2.Developing business processes and storing them away gets mixed reactions.
a.Many processes are defined by and governed by the existing productivity systems and in any case may even have no scope for change, so what will you gain from low level process views?
b.The speed of change is such that keeping recorded process up to date will be more of a chore than a benefit in many businesses.
c.If you can set this at a high enough level such as business goals and strategy it is very useful, but then the benefit depends on the audience who will access it, or ignore it. The jury is out.
d.The right place in my view for business process analysis is Opportunities and solutions phase and the key to this is JIT (as per the Agile principal) if you want relevance and accuracy. In my opinion,  nobody in their right mind would invest a large sum on a solution based on old archived process diagrams at least not any experienced business analyst.
e.Finally new systems generally herald a new process, or new approach to the goal and therefore brand new processes. As-Is modelling is only an exercise to extract the business rules in order to build a better process against them, otherwise, who needs a history level in a manic business world?
3.I don’t see a Zachman like illustration that lets a business stakeholder engage with it on a business level without struggling with the gibberish. If I am right about this, then it could be a downfall, because it will end up just a techie thing behind the doors of IT and the business stakeholders look skywards at the mention of it. This is very common and would negate most of the effort.
You only have to attempt to get some sense out of the Open Group website to see what I mean. If reluctantly, at least I was able to suffer it and get a glimpse of the big picture, I can’t see anybody form a business background touching it with a bargepole. I think the implementation needs to be aware of this and cover up for the shortfalls.
4.It’s a techie invention and although pretty good once you uncover it, it was “not invented here”.
5.The Migration Planning function is a potential route to serious friction. Modern business won’t stand for an IT function that wants to “wag the dog”. When they want it in three months because it is important, no amount of processes will stop them and I have seen many big rows break out over this. I recently saw a head of projects and a director resign within two weeks of each other over rows based on this function.
ADM
The process looks something like this:
The colour coding is a suggested classification based on my interpretation and my own experiences, but I fully expect to be contradicted as I did spend a only a few hours on this and concluded the following:
Pale Blue box – CTO/CIO/Technical architect and/or Enterprise architect.  Plenty of sign-off seems to be the order of the day.
Black box Technical architect’s role but very much in collaboration with the IT function, who will have to implement it and keep it running. Often the ITIL process sits between the black boxes and the rest of the process.
Red box  – Business focused business analyst and I would say a lot of verification needed to make sure the techies don’t build a toy when nobody is looking.
Green box  – Data analyst and systems analyst
Purple box –  Business analyst role working with the business to identify opportunities, develop business cases and priorities shortlists
Blue box –Test, or QA manager and UAT signed off by  Business Analyst and UX manager.
As a rough guide the phases look something like this:
Framework and principals
1.define the scope of the project,
2.adapt the ADM process to your situation
3.identify constraints, document the business requirements, and
4.establish high-level definitions for both the baseline (starting) architecture and target (desired) architecture.
Business architecture
1.business modelling,
2.highly detailed business analysis, and
3.technical-requirements documentation.
Information systems architecture
1.develop baseline data-architecture description
2.review and validate principles, reference models, viewpoints, and tools
3.create architecture models, including logical data models, data-management process models, and relationship models that map business functions to CRUD (Create, Read, Update, Delete) data operations
4.select data-architecture building blocks
5.conduct formal checkpoint reviews of the architecture model and building blocks with stakeholders
6.review qualitative criteria (for example, performance, reliability, security, integrity)
7.complete data architecture
8.conduct checkpoint/impact analysis
9.perform gap analysis
Technical architecture
The infrastructure necessary to support the proposed new architecture.
Opportunities and solutions
Evaluates the various potential solutions and prioritise the ones that deliver most.
This is about business cases and stage gates and good financial analysis, but it is also about good business instinct and gut feel and requires business people to make the decisions in possession of the best possible information.
Migration planning
This is mostly programme management in that it involves acting on the outputs of the previous step to plan the flow of project through system taking into account opportunity and dependencies.
Implementation governance
This is most closely aligned to QA functions in project and programme management
It sets the acceptance criteria
Architectural change management
This is about updating the change management documents with details of new additions. For me, this is better handled by ITIL and as a PM this normally means an agreed handover to IT as part of the implementation and rollout plans.
Once in place
After the first implementation, I interpret it that the business will engage with the Opportunities and Solution function which triggers a quick iteration of the previous steps of ADM to make any required adjustments and the new solution progresses through the remaining stages.

TOGAF

I couldn’t find a readable short summary anywhere so I decided to bear the pain, skim read it and write a summary for any unfortunate person trying to get a quick overview.

My conclusion after a few hours of wading through the treacle is that it is well put together, if horrendously poor in it’s presentation.

A slight contradiction is that parts are already a bit outdated in that it pays too much attention to things like portability in a world where interoperability is the buzz word and the “cloud” is the modern metaphor, nevertheless any slight shortcomings can easily be overcome by intelligent people using it.

A sharp warning though is that; even more so than the ubiquitous Prince2 for project management, TOGAF won’t make an architect of you, you have to bring all the skills to the table and then it helps you get things in the right order, but even that is up for interpretation, so you really must know your stuff and have integrity. It is not therefore a reliable hiring tool in it’s own right.

TOGAF calls itself a Framework, but the main body of it is ADM which is in fact a process for doing architecture.

It follows the usual pattern of breaking up enterprise architecture into four logical steps (steps because it is a process)

The technical architecture uses a method closely allied to the idea of inheritance in OOD whereby the patterns and principals are inherited from more generic ones and you then build something more specialised on top of all that.

E.G.

Organisation specific ( This is the innovative bit)

Industry specific ( This is common to all organisations in a specific industry and saves a lot of time)

Common Architecture ( This varies little from one organisation to another therefore requires a little more effort than foundation, but saves much effort)

Foundation architecture ( This doesn’t vary much from one organisation to another therefore requires little, or no effort)

They call this idea the “Enterprise continuum

Questionable ideas

Things I would question:

1.The Foundation architecture specifies knowledge bases that you need to develop such as a set of standards and a Technical reference Model. This part is potentially futile in that:

a.constant change often makes last years standards out of date, but nobody ever updates these things

b.notwithstanding the previous comment, experience suggests that few people ever refer to them even in some big name software houses, so only do it if you can enforce it.

2.Developing business processes and storing them away gets mixed reactions, I suspect that this should be built up over time as and when you have occasion to look at particular areas of process.

3.I don’t see a Zachman like illustration that lets a business stakeholder engage with it on a business level without struggling with the gibberish. If I am right about this, then it could be a downfall, because it will end up just a “techie” thing behind the doors of IT and the business stakeholders look skywards at the mention of it. This is very common and would negate most of the effort.

You only have to attempt to get some sense out of the Open Group website to see what I mean. If reluctantly, at least I was able to suffer it and get a glimpse of the big picture, I can’t see anybody from a business background touching it with a bargepole. I think the implementation needs to be aware of this and cover up for the shortfalls.

4.It’s a techie invention and although pretty good once you uncover it, it was “not invented here”.

5.The Migration Planning function is a potential route to serious friction. Modern business won’t stand for an IT function that wants to “wag the dog”. When they want it in three months because it is important, no amount of processes will stop them and I have seen many big rows break out over this.

ADM

The process looks something like this:

ADM process
ADM process

The colour coding is a suggested classification based on my interpretation and my own experiences, but I fully expect to be contradicted as I did spend a only a few hours on this and concluded the following:

Pale Blue box – CTO/CIO/Technical architect and/or Enterprise architect.  Plenty of sign-off seems to be the order of the day.

Black box Technical architect’s role but very much in collaboration with the IT function, who will have to implement it and keep it running. Often the ITIL process sits between the black boxes and the rest of the process.

Red box – Business focused business analyst and I would say a lot of verification needed to make sure the techies don’t build a toy when nobody is looking.

Green box – Data analyst and systems analyst

Purple box –  Business analyst role working with the business to identify opportunities, develop business cases and priorities shortlists

Blue box –Test, or QA manager and UAT signed off by  Business Analyst and UX manager.

As a rough guide the phases look something like this:

Framework and principals

1.define the scope of the project,

2.adapt the ADM process to your situation

3.identify constraints, document the business requirements, and

4.establish high-level definitions for both the baseline (starting) architecture and target (desired) architecture.

Business architecture

1.business modelling,

2.highly detailed business analysis, and

3.technical-requirements documentation.

Information systems architecture

1.develop baseline data-architecture description

2.review and validate principles, reference models, viewpoints, and tools

3.create architecture models, including logical data models, data-management process models, and relationship models that map business functions to CRUD (Create, Read, Update, Delete) data operations

4.select data-architecture building blocks

5.conduct formal checkpoint reviews of the architecture model and building blocks with stakeholders

6.review qualitative criteria (for example, performance, reliability, security, integrity)

7.complete data architecture

8.conduct checkpoint/impact analysis

9.perform gap analysis

Technical architecture

The infrastructure necessary to support the proposed new architecture.

Opportunities and solutions

Evaluates the various potential solutions and prioritise the ones that deliver most.

This is about business cases and stage gates and good financial analysis, but it is also about good business instinct and gut feel and requires business people to make the decisions in possession of the best possible information.

Migration planning

This is mostly programme management in that it involves acting on the outputs of the previous step to plan the flow of project through system taking into account opportunity and dependencies.

Implementation governance

This is most closely aligned to QA functions in project and programme management

It sets the acceptance criteria

Architectural change management

This is about updating the change management documents with details of new additions. For me, this is better handled by ITIL and as a PM this normally means an agreed handover to IT as part of the implementation and rollout plans.

Once in place

After the first implementation, I interpret it that the business will engage with the Opportunities and Solution function which triggers a quick iteration of the previous steps of ADM to make any required adjustments and the new solution progresses through the remaining stages.

I remember having very similar thoughts about Pricne 2 when I first tackled it, so I wont be put off.

Perhaps I am one of the stronger proponents of Enterprise Architecture as a means to avoiding meltdown, avoiding waste and keeping the IT funciton focused on value added activities. For this reason I have decided to suffer the pain and see this through.
Maybe I will then get the courage to write a consumable version of this to win back the many talendted and desperately needed people who surely must have been alienated form the profession by the manner in which this is presented.

 

Give big data the heave-ho for now and get a birds-eye view of your business without a single nosql database.  

Previuosly

Good management strategy and practice supported by intelligent modern systems

Lernaean Hydra, your time is up.

Two question must be answered up front :
1. What value does your business delivers best?

2. What do your customers think of it?

Let me explain. One enduring rule of business is “find a hungry mob and feed them” and the other one is “stick to what you do best”.  You’ve probably spotted the potential for a mismatch here. I am assuming that when you find that hungry mob you are very good at feeding them. We are just talking about things like menu, ambiance, pricing and how you operate in the kitchen and waiting departments here. We are also very aware that value is in the eye of the beholder, that is why we are asking the second question.  The answers to these questions are deeper than you may have thought and understanding the cause and effect relationships within the business requires an organised approach, yet you’ll be surprised how simple it can all be made.

 

Lets begin with; what value do you deliver and where it comes from.

Perhaps you believe you know the answer to this; We provide widgets

For the sake of argument: You are a Gas supplier and you are relatively successful but you have known for some time that you are at risk from bigger chains and you need to differentiate and to up your game. Your profit margins are less than they might be.

Lets take the Porter view. ( “Competitive Advantage” Michael Porter ) First we break the activities of the business into Primary and secondary activities. Primary activities are engaged in providing value to the customer e.g.  Receiving and distributing internally -> Making stuff-> Delivering -> marketing and sales -> After Sales Service. This is the coal face. Each step, even Marketing and sales adds some value for the customer and your business is able to capture this in profit after paying for the cost of delivering it.

Secondary activities are engaged in helping run the business, e.g. HR, IT, Finance, etc. Trimming and improving secondary services can improve bottom line, but it will have minimal impact on the core of a business according to the theory, but when you are able to see the direct links between supporting services and the value or lack of it that you deliver to the customer and your costs incurred, then you may look at some of these secondary activities in a new light.

Furthermore, it is clear that since Porter’s book “IT has moved to straddle the two categories and then onwards to dominate the Primary classification for an increasing number of businesses. (Successful ones)

 

Porters strategy firm “Monitor” went out of business in 2012 unable to pay it’s debts. To quote Steve Denning in Forbes it was “killed by the dominant force: the customer“ The model below is focused primarily on the customer.

1. Each primary step in the customer journey needs to be understood in terms of the actual activities that deliver value for the customer and in this way you will be able to quickly see what adds value and what is letting you and your customer down. Value for a customer is seen in terms of what the customer is doing and what that customer is thinking, feeling and experiencing while in the process.  Remember, that experience is very important or even crucial part of the value proposition with both positive and negative propensity.
2.  It can be remarkably enlightening to see the customer journey alongside of the disconnected systems, functions, departments and cultures within your business.

3. No solutions of any value can be outlined until the true impacts and interrelations of actions are understood

 

Below is a combination of examples from actual projects I completed in various sectors, but I have changed some of the details and left out a lot to hide the client identity, only including enough data to demonstrate the principal.

For arguments sake let’s assume the client was a Telco and the product area was domestic mobile phone packages in the UK with the usual blends of calls, texts, internet and hardware. In the background to this small piece of customer focused work, there was a backroom battleground involving the shut down of call centres resulting in the resignation of a C level executive, firing of a senior manager and outsourcing of remaining call centres to India in an attempt to reduce the cost of customer service. There was a hot potato being bounced back and forth and where it fell it was felt.

 

For the sake of argument let’s say the customer lifetime was 4 years and the cost of customer acquisition was £240.00. Average revenue per customer was £600 a year. Immediately you can see that customer acquisition cost 10% of revenue.
-Each year a customer remains the annual profit margin on that customer increases.
-Another way to look at this is: every time a customer needs help and ca’nt get it there is a 25% chance they will leave and the cost of replacing her will be £240.

That means £240 * 25% = £60.00 down the pan for each frustrated customer.

Bear in mind now that reducing costs in order to declare increased earnings was the main goal of the business at this point in time. Perhaps you already see an anomaly here?
Some research done with a small number of focus groups and some with customer services people

demonstrated very clearly that:

a) Customers shopped around as their current contract came to an end and were mainly driven by the best handset they could get and often focused on the current most popular handset.

b) Many customers were on mailing lists of competitors who knew when the contract would end and mailed them with offers ahead of this date. Good offers.

c) Customers who had received unexpectedly large bills recently (last three months) were much more likely to leave, they hated surprises even more than they disliked the nuisance of changing supplier.

I proceeded to map some of this information in context as per example below.

Customer Stage  Forward planning Researching online and stores Purchasing Organising for receipt of product Getting familiar Dealing with issues Renewing Changing package Leaving
Doing Talking to friends/ keeping abreast Browsing shop windows/ websites Looking for best deal, least hassle, warm feeling Waiting for delivery Struggling to learn new interface and get all songs and contacts across Getting help with complexsettings Responding to offer of a replacement handset Calling to get help changing Demanding a PUK code
Thinking I always like to have a phone I can be proud of in my peer group I wonder if I could be doing better I don’t want to find next week that my friends are paying less or getting more I hope everything arrives as planned so I am home to receive it I must get used to the core stuff quickly so I don’t get in trouble. Well soon find out if that annoying advert was a lie, I bet they cant be bothered to answer the phone even -I should be shopping around but the others are probably just as bad and anyhow -I don’t have time right now,I cant wait to get out
Feeling I need to always  carry the right status symbols Insecure Nervous and unsure, in need of reassurance excited I love opening an attractive package that’s presented well Disappointed with the level of support, but not really surprised.I’m frustrated with the difficulty of finding what I need on this website.I don’t have time for this.Angry about a huge bill with no explanation and confusing bill – Resigned to being let down, or- Determined to leave at any costFeeling of power and sometimes an appetitive for revenge Confused, by packages, apprehensive
Experience Mildly amusing 

 

Interesting but sometimes overwhelming A nerve wrecking experience for most Positive Frustrating for some, annoying for many,Very negative if no help can be had Generally a negative experience, sometimes very negative Positive mostly especially when they are a new customer having just left another supplier balanced Positive and expecting better things with new supplier
Business function and location Web services- LondonRetail-  North East Web services- LondonRetail- North EastDirect marketing- London Local ShopOnline 

 

Payment provider

Courier -LondonDispatch -North east 

Shop- Local

Manual – ChinaOnline support – Outsourced India  Manual – ChinaCall centre –IndiaShop- Local Call centre-LondonShop- local Call centre-LondonShop- local Call centre –Indiaand 

Call centre-

London

 

 

Systems eCommerceERP(Two separate systems) eCommerceERPCRM

 

Web stats

 

(Little or no integration between these three.

We don’t know if it is the same customer!)

eCommerceERPCloud service

(Payment system is totally external and data cant be accessed)

Cloud courier apiERP{(No integration available, ERP nows nothing about courier progress) (Little or no communication happens here.
Call centre handles own training from the manual.All a bit up n the air.)
Inbound CRM is separate to central CRM and there is no knowledge of these inbound calls in other parts of the business for the most part The local call centre is trained to retain customers when a call is sent to them. Their conversations go in the CRM and others are aware of it The local call centre is trained to retain customers when a call is sent to them. Their conversations go in the CRM and others are aware of it Calls are transferred to retention team who try to bribe them. No analysis of this ever happens afterwards

Table 1

 

 

We decided that we needed more detail in some key areas to support our goals and help us solve issues we had identified through the web stats, the CRM and the eCommerce data.

First we looked more closely at the purchasing cycle and here is a representation of what we found.
Purchasing

 

  1. We have different personas who differ somewhat in their behaviour, but a safe catch all is to say that our customers never stop shopping around for alternatives. Within hours of getting the product they are telling friends and getting comparisons.
    Often they find that they could have done better elsewhere, sometimes via our own channels. When this occurs there is major loss of trust by an existing customer, but when others do it, it drives potential customers to us.
  2. The desire to change product, upgrade, or move may also begin when a new must-have feature hits the market and the customer’s friends are getting it. This can sometimes happen within days.
  3. Loyalty to the actual product brand is strong but loyalty to our brand is weak, we have no kudos like a product maker has. They can have the next version of their favourite cult phone, but still find a new provider.
  4. It was difficult to find any real logic behind most purchases, fundamentally the customer fell in love with a handset and then needed evidence to justify this purchase and the price they would pay. The fear was hidden costs, surprise bills, or looking foolish when their friends got a better value deal.
  5. The channels were confusing them, they saw deals online, but when they went to our shop, either it was not there, or it cost more, or sometimes less. They couldn’t understand this. Once we were losing heavily to an online competitor who undercut us by just £1 a month, wile we had a better deal hidden away in many of our shops. Nobody could explain that.

 

Here is the journey simplified

 

  1. In contract but keeping abreast of new stuff and new deals
  2. Close to time for a new deal and really researching online and asking friends and social media connections ( 20% will call or to cancel )
  3. Nearing decision time, has a strong preference that she can’t really afford and a second preference and shopping around for the best possible deal
  4. Decision time and now in a blind panic. What if it is cheaper next week?, Should I go for the cheaper of the two or indulge myself?
  5. Trying to place the order and the quirky ecommerce system keeps complaining and asking her to re-enter stuff. (38% leave at this point according to our records –in fact we know the precise page and we have found out why)
  6. Order placed and still wondering if she should cancel before it arrives (Another 12% will cancel)
  7. Package arrives and enjoying the quality of the packaging and the beauty of the new phone. Upset by the dreadful manual and having to learn a whole new set of menus etc. ( 30% end up calling in for support and waiting up to 15 minutes before spending a further 40 minutes on the phone)
  8. Nervously transferring data and hoping not to lose anything ( Only 5% now lose data and support can usually help them)
  9. Ok happy customer. Along the way we lost 70% of those who started and made ourselves very unpopular with about 20% of the remainder i.e. 6% overall.

 

Causes of Churn

In simple terms, although there are several problems identified, we found three causes of people calling in to end a contract.
i. The 6% we annoyed by giving a poor experience when they received the last product.

ii. Receiving unexpectedly high bills that left them suddenly short of cash at the end of the month. This caused particular and lasting annoyance, especially when the bill was incomprehensible.

iii. The 18% who called customer services and had very poor experiences ranging from incompetency, to occasional lack of sympathy to confusing IVR and unacceptable delay times.

In addition, although we couldn’t quantify the affect, we found those who had bad experiences talked about it publicly in social media while the others were silent and the overall appearance was of a lot of unhappy customers. This was very imbalanced and clearly damaging.

To keep this simple and get the most holistic view, we used two problem solving techniques to get to the bottom of things and prioritise our actions.

 

Problems  don’t happen in isolation, nor do solutions act in isolation, therefore a holistic view is a critical starting point. Below, I recount two simple techniques I frequently use when solving systemic problems like those we discussed last week <<Click here to open last weeks instalment in a new window>>

 

We identified problems and sub problems:
The technique is simple; ask why and to the answer ask why again. Do this five times before you accept the explanation. At this point you will have a new insight and abetter understanding
Propositions struggling to compete and losing grounds.

 

Caused by:

Positioning on price point + struggling to compete on cost

High cost per customer = low margins

i.e. We could accept low service, or we could lose in the battle on price

margins

We drew an affinity diagram to track the interconnections between these problems and sub problems.

Pretty early on it became evident that the quality of the customer experience was core to the customer churn which in turn had impacts not just on customer numbers, but subsequently on our ability to offer a competitively priced proposition. A classic vicious circle.

diagram1

Diagram 1

 

To focus on the key aspects and explore the causal route a little more we took the key causes out of the affinity diagram and used them to build a cause and effect illustration.

As you will probably notice this helps remove some of the noise and focus in on more critical factors.

The relative  importance of the various factors was tested with high level business cases that demonstrated in very clear terms what the financial implications of each issue and potential negative impacts where relevant.

 

The result looked like this;

diagram2

 

 

Diagram 2

 

The figures demonstrated clearly that improving the product and delivery especially at Moments of truth and   becoming customer focused would reduce the traffic to the call centre sufficiently to counter balance the cost of a decent service and still improve the margins enough to give us the option of being more competitive.

Problem 1  High costs
What our research also told us was that with an improved service and happier customers we would be under far less pressure to compete with the cheapest competitors,

Problem two dwindling sales

 

The second primary cause of High customer acquisition costs and dwindling sales was weak marketing

Our research into the causes of weak marketing highlighted a systems issue. In simple language, marketers had insufficient access to key customer data and live systems like ecommerce and “myaccount” lacked the information needed to offer the right help to customers at the time they needed it i.e. when they were on our site seeking it.

 

The underlying causes was totally technical. The entire business had been migrated to run on SAP at a cost of obscene numbers of millions and reputations were at stake so no hint of deference could be entertained. Frankly it didn’t work and this was mainly down to being implemented without full understanding of the business need.

Within a year of implementation it became clear that accessing SAP to get basic information about a customer for another system such as an eCommerce site cost an unbelievable 1million plus per interface.
Not only that, but the levels of traffic to SAP from a busy eCommerce site would have driven a further investment in licenses and hosting beyond most people’s imaginations.

On top of this round trips to SAP in its current implementation was slow even when available.

We could have a website and customer services with no customer knowledge or fork out millions on upgrading an already obscenely expensive piece of technology.

We needed a way access key  data that was in SAP about our customers in high volume and with very fast trip time in order to answer questions on the customer service desk and to serve intelligent web pages on our sites

Delivering benefits is no longer about getting a system signed off

(Who’s minding the shop?)
This blog is an attempt to stimulate discussion and understanding of the balance of responsibility for delivering business benefits from IT investment.
It is now fairly widely recognised that this is not as simple as choosing a system, getting it working and reaping the benefits, but as yet we can call on very few examples of good practice in making it work.
The business argument.
We have no time for discussions about how it all works, we just want a business case and if it gives us the ROI we need with acceptable risks, we want to do it. At least that’s the CFO’s line. In reality, many of these projects begin life as pet projects of the CEO, or other senior manager and it is clear from the outset that the question is not will it work, but make it work.
This is largely an understandable attitude, because businesses must be directed and managers must manage and sometimes the outcome can be a life or death one.
The IT supplier argument.
Be the supplier an in house department or an external supplier, the situation is not hugely different. The supplier tends to confine their responsibilities and activities to making the technology work. Better suppliers provide advice and guidance on rollout, but that is the limit of the activity and there is no responsibility beyond meeting technical requirements. This is wholly understandable, because the supplier has no control over whether the business case was correctly put together, whether the requirements were detailed enough or whether the culture is sufficiently flexible or disciplined to make the new system work and deliver benefits.
So who‘s minding the shop?
Well the answer very often is that nobody is minding the shop. Rarely is anybody in the business equipped to do this job and the suppliers are not getting involved.
The most usual scenario is that the system is delivered and there are issues and misfits that become immediately apparent, then there’s a return to the drawing board and changes are made and a re-launch and things are much better. Often you’ll find a new project with a new name the following year, billed as an upgrade and it finally knocks the thing into shape. Two years late and dramatically over budget the project is complete and nobody ever asks whether it delivered benefits because at this point the busines is bus with a whole new set of issues.
The other common scenario is that a consultant is hired to manage the project, he/she is expected to have a project management qualification PMI/Prince etc and have handled one of these type of projects before, maybe even in this industry.
Often the contract is already in place, the product defined and dates and budgets set.
Where does a Consultant go from here?
1. Recommend someone else who needs a bit of experience.
2. Take it on and hope for the best
3. Challenge the client at interview and get passed over as wrong attitude
4. Take it on and then challenge the client later via a robust consulting methodology
5. Challenge the client at interview via a robust consulting methodology
6. Offer to challenge the project’s soundness for a reasonable fee and provide a dispassionate report.

The Bridger

“Bridger” A person who acts as go between or translator between the management of a business and it’s IT department or IT partners.

The background

Since the arrival of computers in the enterprise there has been an unfortunate and counterproductive power struggle between IT departments and other areas of the business.
IT professionals share a great deal with other professions such as law and medicine in that they have their own impenetrable language, they are critically important to their clients and they simply can not be substituted with amateurs.

The difference between IT and other professions is that systems and IT tools, especially systems are right at the heart of the every day operations of the business and a fundamental part of every aspect of the business operations.

Naturally it is a source of very serious concern for business that such a critical function as IT is managed by people who are more often than not detached from the business functions and often lack very much understanding or training in business issues.

The key problems faced by a Bridger

Problem one – The Business versus the IT department

In order to create and maintain a competitive advantage for the enterprise it is essential to combine deep understanding of the business, enlightened leadership and a thorough grasp of what IT can achieve.
The business leadership are custodians of business strategy and direction and hold the deep understanding of the business.
The IT department or partners hold the deep knowledge of how existing systems work, the impact of changes to them and what new technology is capable of.

Almost without exception, the two areas create and maintain their own plans and strategies in isolation and believe at some level that they can exist successfully in silos. This standoff when it occurs to any degree, is destructive and robs industry of billions annually in lost opportunities and wastage, mostly surfaced as failed projects.

How the Bridger can help

In the short term, a Bridger can mediate between IT and the business translating between IT and Business language and negotiating win/win agreements.
In particular the business needs to understand impacts and costs of sudden demands for change on the IT infrastructure.
The IT department needs understand clearly the real life drivers that have potential to place unwanted demands in their inbox unexpectedly. Speaking both languages and understanding the underlying problems on both sides makes the Bridger ideally paced to forge this understanding.

In the medium to long term, a Bridger can help forge an understanding by helping to establish sustainable frameworks that see joint plans and strategies developed by the business and IT jointly to forge new partnerships in place of the silos or misunderstandings.

Problem 2 – Business analysts are trained in IT

The most important element of every software project is what it delivers to the business. How clever the technology is, whether it was delivered a little early or late or slightly under or over budget, are all relatively unimportant in the grand scheme of things, but if the system fails to deliver value to the business, it will be a costly failure.

Whether or not the system delivers value is down to understanding the needs of the business, optimising processes for automation and implementing the accompanying cultural change to make it work. These are all business and people centric skills.

This critical job normally falls to a business analyst, but there lies the problem.

With a very few exceptions, business analysts are not trained in business analysis or indeed in any other business discipline. The majority are programmers who went down the analysis route and see their role as designing systems and working with IT delivery. In recent years some have been emerging form universities with degrees in business analysis, but even then they are quickly sucked into the existing culture and still lack the business knowledge to understand the issues and the gravitas to engage senior management.

It’s hardly surprising then that there are frustrations, communication difficulties and disappointments as the enterprise attempts to manage its it investments.

The alternative to this business analyst route is when the business, in exasperation appoints someone from the business to gather requirements. This is an even bigger mistake because the non it person has no idea how to extract or record requirements or how to communicate them to IT people and especially he/she has little chance of monitoring the accuracy or quality of the deliverable.

Another very common error seen more and more in recent times is to react to this problem by appointing a non IT trained Project Manager to deal with the business and systems sides of the project. While business stakeholders may enjoy better communication in the short run, their holiday is generally short lives as this strategy lets IT off the hook and leads to a catalogue of poor decisions and a high proportion of failed software projects.
Government in particular are guilty of this mistake as they place too much trust in partners and maintain little, or often no internal skills and knowhow to oversee these huge projects.

How the Bridger can help

The Bridger is by nature a business person with the ability to explore, understand and improve processes as well as the training to explain and present them to the IT department, or to work in tandem with the IT department making sure that they get the business viewpoint right.

This approach ensures that the business goals are represented by the new proposed system and the level of change is understood and accepted by users so that a system can be designed and implemented with confidence.

Problem 3 – Stakeholders, Users and Technologists, the tradeoffs.

Stakeholders (define here as the people providing the investment and expecting the rewards in productivity) are all powerful within the project team and all things considered, this is how it should be.
Users, even for an in-house system, are still key. Unless you understand their needs and serve them, the system will not deliver value.

Technologists are the people who know how to deliver on your needs and to make it reliable. Nobody notices them when everything is running smoothly, only when it goes wrong so you can’t do it without their help and goodwill.

When it comes to defining the final feature set of a system, stakeholders will have a detached and uninformed view of what happens on the shop floor and will want to dictate how the system should be. If they get all own their way it will almost certainly be a disaster.

Users will resist change and will not want any new system so their feedback important as though it is must be understood in context and then interpreted at the right level to deliver achievable changes.

IT will want to dictate the system based on what fits comfortably with that they already have and will have no empathy with users or stakeholders.

How the Bridger can help

The Bridger can help by forging and maintaining the right flows of communication between IT, Users and Business so that ultimately the Business gets what they want through ensuring that IT works with the User to maximise process and pitch cultural changes at a level that is achievable and lends itself to a successful implementation.

Ed Taaffe