Who do we really measure for

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

This is the first principle behind the agile manifesto. Most organisations claim to be customer focused and driven by the needs of their customers. But are they really? It is interesting to look at the measures these organisations use. Who do these measures actually serve? Do they answer the question are we satisfying our customers, or do they provide management information?

Dr. W Edward Deming is often misquoted as having said, you can't manage what you can't measure. Managers, who devise increasingly inventive measures to help them perform their management activities, often adopt this as a mantra.

What Dr. Deming actually said:

the most important figures that one needs for management are unknown or unknowable (Lloyd S. Nelson, director of statistical methods for the Nashua corporation), but successful management must nevertheless take account of them.

Its comfortable for managers to believe that measurement exists for their benefit, after all it’s a good way to prove their own value. It also means that they can avoid those uncomfortable things that are difficult to measure.

If customer satisfaction is your primary purpose your measurement has to begin and end with the customer. The first stage of this journey is to understand your customers’ needs and desires.  Surprisingly this is not always as easy as it sounds.

What does the customer want?

Konica wanted to increase their market share in the consumer camera market. The feedback that they received from their customers only suggested minor improvements. That is until Mr Takanori Yoneyama made this suggestion: Perhaps we are asking the wrong questions. We ask for feedback on our camera. But people don’t purchase our camera in order to own a camera. They buy a camera in order to take pictures. We see ourselves as manufacturing and selling cameras. Customers see us as a source for acquiring the ability to take pictures. Perhaps we should start seeking feedback on the pictures.

This approach proved to be a turning point in Konica’s history. Some of the pictures they reviewed with their customers were terrible. Poor focus, incorrectly exposed or superimposed images. These customers had a common theme. “Your camera is great, but I’m not a good enough photographer”.

Konica used this feedback to develop cameras that did not rely on the skill of the photographer to take good pictures. Today we take auto-focus, auto-exposure, auto-flash and pretty much auto-everything for granted in cameras. Would this be true if camera manufacturers had kept asking, how do we make a better camera instead of how do we help our customers take better pictures?

Once we understand what our customers want from us there is an obvious question to ask. Can we do it? In other words do we have the capability to meet our customer demand? If not, what needs to change in order to do this? If we do have the capability is it appropriate? A manufacturing company that scales to produce 10000 units per month when its customers need 100 units per month will probably fail to meet the value for money need of its customers.

Are you wondering why I assume it’s the customers’ needs that are most important? Quite simply because they are. If you believe you are in it for the money or your company’s real purpose is to maximise shareholder value there is usually a much easier way of achieving this.  Businesses who claim to focus on the customer in order to meet their true objective of making lots of money are likely to be less profitable than those whose real objective is satisfying its customers. If shareholder profit was the only consideration a very effective way of achieving this may be to sell the assets and close the business.

It is only when we understand our customer needs and the capability that is required that we should start looking inwards. It is during this inward look that we will look at our own processes. We will uncover bottlenecks and inefficiencies by keeping focused on the customer needs. We should ask How does this process help us to meet the customers needs? rather than how can we improve this process?  Discovering that the answer to the first question is it doesn’t means that we can get rid of an unnecessary process which will result in both cost savings and increased efficiency. If we had asked the second question we may have spent a lot of time, effort and money on trying to improve a process that doesn’t contribute towards delivering any value to our customers.

In our inward look we are really only striving for one thing. That is to have the ability to consistently and reliably meet our customer demand. If we can achieve this we will have improved efficiencies with higher quality as a result. If we focus on quality or costs these become measures and goals in their own right, which become separated from the customer needs. We may achieve these goals and alienate our customers because we haven’t taken their needs into account. Would Konica have increased its market share if it had focused on cost savings or quality?

A solution that focused on sales figures may have been to reduce the price. If they had adopted this approach they would have reduced their profits so jobs would have been lost. Quality would have suffered and sales may have declined further, especially if their competitors had been asking the right questions of their customers.

The shareholder value approach might have seen them reducing their staff by 30%. (Sales are down 25%, if we cut costs by 30% our profit will go up and the shareholders will be happy). This is of course a very short term solution. Konica did increase its market share by innovating and gained more customers, and more sales, along the way. If they had followed the cut costs approach their employees would have been so busy trying to keep up with current demand that innovation would never have happened, and quality would have suffered. Sales would have continued to decline until it reached the point where the company was no longer viable.

Once you have made changes it is important to verify these with the customers. Internal measures such as sales figures, complaints volume and customer retention many give a useful indication, but ultimately it is the customers opinion that counts.

Measuring for improvement

It is important that the measure for improvement actually measures that you have improved in the area that you were aiming for improvement.

A company that provided a data analysis service received numerous complaints that the results of its analyses were taking too long to deliver. The CEO met with a key customer after yet another late delivery. One of the comments made by the customer was what really annoyed us was that you told us that the work would be completed in 4 weeks and then delivered after 6 weeks. If we had known that it would take 6 weeks we could have planned accordingly.

The CEO returned to his office and announced that customers were unhappy at the accuracy of estimates. He immediately created targets and set incentives for improving the accuracy of estimates. The next job performed was completed on schedule. In due course the estimates were consistently accurate and bonuses were paid. Job done?

A few months later the CEO called the same customer when he noticed that there had been no orders for several months.  Naturally he asked why this was. Simple, said the client. You take too long to deliver our orders. Thank you for improving your estimates because now at least you are being honest about the fact that you can’t meet our needs.

This shows how easy it is to avoid measuring the thing that actually matters to the customer. Although it seems obvious, if the customer is unhappy about the time elapsed between placing his order and receiving the results this is the only meaningful measure for the customer.

Just as important as measuring the right thing is measuring the whole thing. Assuming that improving the estimation accuracy will improve delivery time was a mistake, even if we ignore the fact that the CEO jumped to an invalid conclusion on the basis of a single conversation with one customer.

When the CEO did measure elapsed time between order and delivery he discovered that it took an average of 3 weeks to process an order. This meant that the work was not started for at least 3 weeks. Fortunately this was easy to change and the company was able to improve its customer satisfaction and win back lost customers.

If you are intending on improvement it is important to start with a purpose. Creating and taking measurements will create additional work. Why would you want to do that? As we have seen from the above case study it is important that you understand how the whole system will be used and who will benefit from it.

Once you understand the purpose and what needs to be measured pick a priority measurement target. Identify the purpose of the process that you intend to measure and what the measures are. To help in identifying what the measures are think about the following questions:

  • What do I want to learn?
  • What will the data tell me?
  • What questions do I need to be answered?
  • How will I use what I learn?

You also need to understand how your process and measurements fit into the system as a whole – remember the estimation accuracy!

It is useful to check your definitions of what you are trying to achieve and measure. You are less likely to go wrong if you think of these from your customer perspective. If your customers want a faster delivery time what does this mean to them.

The concept of delivery time means different things to different people.

  • The delivery driver measures from the time the product is loaded into his van until it has been delivered.
  • The despatch clerk measures from the time he produces a shipping note.
  • The accounts office measures from the time the order is processed and payment approved.
  • The customer measures from the time he places the order until he has the product in his hand.

You should also define how you would recognise if things have improved, deteriorated or stayed the same. Don’t forget that this is in the context of the purpose. Improving estimation accuracy did not help the customer receive his product any faster.

There are a few things to think about before you start collecting the data. Some of these are obvious, such as how to gather the data, how to ensure its integrity, what technology is required to analyse it as well as what training needs to be conducted.

Others may be less obvious. How will the act of measurement affect the behaviour of the people being measured? Will they be suspicious of the reasons for the measurements, e.g. management doesn’t trust us, or are they trying to work out who to get rid of? Remember that you are trying to improve the performance of your system by improving the processes in it.

It makes sense that everyone involved understands the purpose of the measurement and realises that you are not out to get them but are trying to assess the current reality. This will also help protect against deliberate or subconscious efforts to influence the measurements. This will usually not be done maliciously. Human nature is to try and please. If somebody knows that you are measuring something they will automatically try to anticipate the desired result and strive to achieve it, or to prove that they are already doing it.
Once you start collecting your data start on a small scale. This allows you to identify the flaws and minimise the disruption before moving to a larger scale. Don’t forget PDSA. Plan, Do, Study, Act. Repeat this as often as you need to. Once your measurement system is in place you should standardise it to make sure that it continues measurement in a consistent manner and then move on to the next measure using the same approach. Once all your measures are in pace continue the PDSA cycle. Measurement is time consuming and disruptive. Organisations have a tendency to stick with their measures long after they have forgotten why they started measuring in the first place. Some measures are worth keeping. Others reach a point where they add no further value as you stop learning from them. If this is the case stop taking them. They are of no further use to your customers.