Myths of Measurement: Perfect Employee Measures Exist

The full contribution of an employee can’t be measured with one metric. Over the last decade, I’ve asked thousands of HR professionals—including some very sophisticated ones—this question:

Can you give me an example of a measure that represents the total value an employee adds to the organization? 

No one has ever been able to do this. Sometimes they suggest a measure from a different company or a department that they don’t understand well. We want to think that somewhere, if we look hard enough, we’ll find the perfect measure of employee performance—it’s just that our own department or company is structured in such a way that we can’t do it.

Admittedly, this is a trick question. Everyone knows perfection doesn’t exist.

We want to think that our performance measurement system is really good—after all, we use it to make all sorts of decisions, including pay. On reflection, however, it’s clear that any measure of employee performance is deficient in that it doesn’t capture the entire value that the employee contributes. Any measure is contaminated by things beyond the employee’s control. Finally, any measure of an employee’s contribution is imprecise in that it isn’t really capable of making precise distinctions, either between people or between different aspects of a person. There is always an element of error, statistically speaking.

A good example is sales performance. Total sales may seem like a perfect measure until you consider the many factors that influence the measure. Any measure of total sales will be deficient because it doesn’t represent the good will associated with honestly representing a product. “Total sales” is short-term. Good will affects future sales. Every measure is deficient in that it doesn’t tell the whole story.

As a measure, “total sales” is also contaminated by factors beyond the sales representatives’ control—factors such as marketing, competition, and the quality or value of the product. Any seasoned sales representative will tell you that the territory will affect a salesperson’s ability to sell. In addition, all measures are imprecise. Consider two sales people: Both had a great year, and their annual sales were within a few thousand dollars of each other. Can we really differentiate between the skill, effort, or success of the two? No.

In practice, the myth of a perfect measure leads to misunderstandings about performance and capabilities. Here’s an example.

A call center manager was concerned about the employee taking the fewest calls. Clearly, this long-term employee was starting to slide. His performance had been in a slow decline.

The manager called the employee into his office. Being a sensitive boss, he asked the employee how things were going and if anything had changed in his life. Finally, he got down to business and confronted the employee about his performance—knowing there were no external factors affecting performance.

The employee’s response was unexpected: “I think I’m totally on top of my game. All my peers appreciate me, and I’m resolving the toughest issues. My productivity is very high!”

After a bit more discussion, the situation became clear. Informally, other call center representatives were escalating the difficult calls to the “problem” employee. If the employee is helping out a team member, he’s obviously contributing both experience and capability. This contribution, however, wouldn’t show up in his productivity measurement data. But his productivity was clearly high, considering that he was handling the most difficult calls.

In this case, the measure was deficient because it represented only quantity, not the quality or difficulty of the task. The employee’s performance was much broader than what was measured. Meanwhile, the other employees’ measurement data was contaminated—their productivity measures were a representation of the more experienced employee’s work. I would also bet this measure is imprecise—that it’s affected by random events, like the call-center computer going down.

This example shows that there can be great value in unmeasured aspects of performance, and that measurement can’t tell the whole story. In the best-case scenario, this call center manager would learn that it’s not possible to rely entirely on a measure that represents only one aspect of employee contribution.

 The Myth of Perfect Measurement in Action 

Organizations often act as if their measures are nearly perfect. As a consultant, I’m often asked to develop measures or provide measure-based guidance on important questions, such as “What value did we get from training?” It’s important to set expectations carefully. Without a perfect measure, there will be no perfect answer.

Early in my career, I spent months, and considerable amounts of a client’s money, searching for a good measure of technician performance for a regional phone company. In one sense, we had very good measures of contribution: the number of repairs completed successfully and the number of phone installations (this was in the era of copper lines and touch-tone phones).  But applying these measures in real-world context, even using strong quasi-experimental designs, we ran into many complications. In the process, my client and I learned, or relearned, a lot:

• Employees "game measures"
• Teamwork matters 
• Organizations are multilayered and complicated 
• Employee performance is multidimensional
• It’s difficult to isolate the contribution of an individual to the organization 
• It’s impossible to perfectly quantify the effect of an intervention (such as training) on an individual’s contribution. 

In the end, we were able to estimate the value of an employee training program. On the other hand, we didn’t use by-the-book analysis, and we certainly didn’t calculate a concrete ROI. Because measures are so imperfect, the ROI analysis would have been based on unrealistic assumptions. We were, however, able to make an educated projection. In the end, that was all that was needed to make good decisions about the training program.

Wouldn’t it be great if there were a perfect measure of employee performance? You could do all sorts of analyses and run organizations in a perfectly rational manner. Unfortunately, the perfect measure doesn’t exist. In fact, I’m not sure you can ever calculate the ROI of a training program, because there is never a perfect measure of employee performance—and that’s what training is supposed to affect.

Beyond this sort of program evaluation and decision making, many talent decisions are based on imperfect measures. As in the example of sales performance above, even sales commissions are based on measures that deficient, contaminated, and imprecise.

Consider More Subtle Outcomes

 In a general sense, phone technicians have clear outcomes: new phones are installed or repaired phones  work correctly. Many, in fact most jobs, have much less clear outcomes.

Consider a teacher in our public schools. Is the outcome of successful teach a grade on a test, success in life or something else.  Measuring the full contribution or value of these jobs is nearly impossible because the outcomes is so complicated. Even if we could define a clear outcome, its measure would be contaminated and deficient.  

What Can We Know? 

Even though all measures are deficient, contaminated, and imprecise, they can still be useful tools to support decision making, if we use them wisely. What’s critical is that we never assume that measures are perfect.

If we make this mistake—if we accept measures as having any degree of perfection—we shut down authentic conversations and prevent the exchange of meaning between managers, employees, and different parts of an organization. In the case of the call center described earlier, for example, relying on a measure without communication, and without the value of human judgment, could have resulted in the organization losing one of its most valuable employees.

Paradoxically, if we accept the inherent weakness of measurement, we can use measures more effectively. When we understand that measurement is an inherently simple approach that will never adequately describe the complexity of employees, or untangle the nuances of employee performance, we can compensate.

One way we can compensate is by relying on multiple measures—while accepting that even this solution is imperfect. Using multiple measures provides the benefit of triangulation, and is often adequate for making decisions, providing feedback, and even supporting human resource decisions.

Ultimately, the weaknesses of measurement provide an opportunity for discussion and learning. Authentic and productive communication that acknowledges these imperfections enhances learning about an organization and its complexities.

Mistrust of Talent Measurement

In many organizations, you’ll find fear and mistrust. You’ll also find that employees often focus these negative sentiments on the organization’s measurement system.

A measurement system won’t work without trust. Trust is the basis of functioning human communication, and as I’ve discussed in previous posts, measurement is best understood as a means of communication. In this blog post we’ll explore these issues, and look at how to increase trust in measurement.

I had a client with a world-class performance management system. The system included many best practices:

• Individual goals cascading from the strategy of the organization
• Employees and managers met at the beginning of the year to discuss goals
• Coaching happened throughout the year, with a formal documentation period
• Final evaluation happened at the end of the year, with ratings of behavioral competencies as well as objective performance metrics.

The  employees hated the evaluation system. Why? They didn’t trust it. This organizations performance management system, like many, was a de-motivator!  

When there is no trust in measurement, the results are often ugly. In pre-revolution feudal France, the physical measurement system was controlled by the aristocracy. Measurement was idiosyncratic, fragmented, and non-standardized; there were approximately 14,000 different units of physical measurement. Mistrust of measurement, or measurement-based decisions (such as those involved in commerce), was one of the factors exacerbating class tensions.

There are historians who report metric riots over measurement issues during this period. When the nobility surrendered their privileges after the storming of the Bastille, these privileges included giving up control of measures.

This may sound extreme, but is it? What if you couldn’t trust that a pound was a pound, and the person measuring your pound of pasta was defining the measurement?

Today, we assume that physical measurements are accurate and standardized—everything from a pound of pasta to land surveys. There are governmental bodies regulating physical measures. We rarely even think about the physical measurements that we use daily.

The same is not true of talent measures. Each organization has its own set of talent measures (for good reason), and these measures are typically controlled by management.

As in the French revolution, those who control the measures have a significant source of power to use and, in some cases, abuse. Before the revolution, serfs often felt they were being cheated by nobility.  They probably were. After the revolution, there was a movement to base measurement on universal, naturally occurring objects. This is the origin of the metric system.

How does this relate to my client with the world-class performance management system? Employees hated the system because it was unpredictable and apparently arbitrary. It was unintentionally failing the fairness test in two ways. 

First, employees saw the performance appraisal procedure as unfair because it was constantly changing. In an attempt to improve the system, senior management was constantly tweaking it. Supervisors and employees weren’t sure how the system would ultimately appear on the intranet. Little was done to communicate how the system worked, or why it was changed. Given the complexities of the measures and the misunderstanding of the process, it’s understandable that employees started to wonder whether something nefarious was going on. Sometimes the tweaks made winners and losers. As a result, a near revolution was brewing in what Elton Mayo would have referred to as a social system. 

Second, employees saw the system as unfair because of the way rewards were distributed. The pool of bonus money was spread among the employees according to the ratings, often as the final ratings were calculated. This is not an unusual profit-sharing plan. Unfortunately, there were last-minute changes, sometimes for departments as a whole, that had unexpected effects on ratings and compensation.

Stability and transparency is needed to make a performance management system trusted and functional. This requires an acceptance of the imperfections of measurement, and much more communication. To a large degree, tweaking will not improve a talent measurement system—it will only serve to further distort the meaning of the measures. Often, tweaks occur because we believe in the myths of measurement:

• My mental model of performance is correct
• Measures are real
• There is a perfect measure
• People and the measurement system do not change in the process of measurement.

Opposition to Measurement

This brings up a second puzzle for a measurement guy like me. Why are so many employees, and unions, opposed to measurement?

Coming out of graduate school, I understood the formal side of measurement very well. I was surprised at the level of animosity toward measurement. I naively assumed that the honesty and accuracy associated with good measurement would enhance relationships, including employee and labor relations. I soon learned that in practice there is little appetite for measures, mostly because of how they have been, and are, used in organizations. 

I’ve found four major reasons for this opposition:

• Employees being held accountable for factors beyond their control
• Arbitrary use of power
• Differentiation between employees
• History of Taylorism.

These four reasons hamper effective talent measurement. They exert such a powerful force that new measurement initiatives start with a lack of trust. Often this lack of trust must be addressed and overcome before organizational learning and employee motivation can happen.

If you know of additional reasons, I’d love to learn about them.

Inappropriate Accountability

As I discussed in a previous post, measures are contaminated by factors beyond an individual employee’s control. For example, low-performing teammates may prevent an employee from reaching her full potential. This is, of course, a fact of life—we can’t control everything.  When employees are faced with a measurement system that ignores factors beyond their control, they can feel that their efforts are futile, and the measurement arbitrary. This tends to erode trust in the organization and management.

Arbitrary Use of Power

Managers have the power to change the measurement system; sometimes this power is used to withhold rewards or punish employees in some way. For example, if a salesperson has earned a large reward based on a previously defined incentive system, sometimes the system is reconfigured to avoid the large payout. Of course, this breaks trust—and it does happen. Unfair actions related to measurement  are remembered for a long time.

In addition, management is often responsible for interpreting or framing measures. Setting unattainable goals, for example, can lead to employee dissatisfaction and decreased morale, as employees give up trying to improve their performance.

Differentiation between Employees

Measurement is often used to differentiate between employees. If you don’t trust the measurement system to make important distinctions related to employee performance, you won’t like differentiation. 

In the coming year, many U.S. school districts will differentiate between teachers using standardized tests. Some will think this is fair, but many will not. Methods have been developed, such as value-added scores, that attempt to statistically control for factors, such as race or wealth, that affect student growth. Despite this, many fundamentally mistrust the standardized tests on which the evaluation systems rest.

Differentiating between employees is also antithetical to unionist philosophy. Unions believe that differentiation reduces solidarity, and they tend to believe that everyone is the same. In fact, this mistrust of measurement, based on valuing both similarity and solidarity, leads to a question often asked by unionists: How can we tell who is better than another? If you can’t answer this question, the only basis for differentiating payment is seniority. While experience does increase knowledge, skill, and even wisdom, it’s not performance. It is, however, an unambiguous differentiator.

History of Taylorism

In a sense, Frederick Taylor’s use of time-and-motion studies to set performance goals destroyed trust in measurement. As I discussed in the last post, Taylor’s approach to measurement was used mechanistically, against employees, and not in partnership with them.

Interestingly, Taylor assumed that employees would embrace scientific management. Sometimes they did, but most often they didn’t. He was, of course, assuming that employees were motivated only by money and the possibility of higher wages based on increased work.  Of course, we now know that this assumption is wrong.

In an interesting turn of history, employee unions and strikes were ultimately the downfall of Taylorism. Strikes by public sector employees caused Congress to hold hearings. After five years, scientific management was essentially outlawed by limiting the use of incentive wages and stopwatches.

Building Trust in Talent Measures

So how do we build trust in talent measures, given that the way measurement has been used in the past creates negative preconceptions? I have three suggestions that will help create a climate of fairness and transparency:

• Make measurement predictable. Build the best technical system you can, and accept that some people will manipulate the system. The solution to manipulation is rarely a better measure—most often, it’s better management and leadership, using the measures.
• Communicate, communicate, communicate! Communication is creating shared meaning. An organization’s definition of the meaning of measures will not be the same as employees’ meanings, or experience. Both meanings need to be acknowledged and managed in a two-way give and take. Never assume that a measure means the same thing to everyone.
• Build trust in measurement. Anything that could be interpreted as using the measures against the workforce or individuals should be avoided.  Apply measurements consistently across your organization. Hold everyone accountable to the same standards, and make those standards clear.  Remember, we want our employees to be thinking about the work and engaging with the organization.  We do not want them to be thinking, “Is this measurement system fair?”