Myths of Measurement: Talent Measures Have No Side Effects

Medication and medical procedures have side effects. Side effects can be harmless (too much niacin can cause blushing) or serious (statins can cause muscle damage). These medications work within the complex chemistry of the human body, and how a person will experience a side effect is unpredictable.

Talent measures are not that different. They work within complex human social systems, and they alter these systems—often in unintended ways. In other words, they have side effects. The side effects most often affect motivation.

There are many ways to increase organizational performance. Many are effective, few are ineffective. And a few can actually damage the organization. Measurement falls into this last category. Because measurement can have negative outcomes, it is worth knowing how these results come about and how they can be managed

When We Measure, We Expect Results

Talent measurement often has the goal of understanding, motivating, or directing workforce performance.  This idea is summarized in the phrase what gets measured gets managed, which is sometimes attributed to Peter Drucker. This phrase is universally framed in a positive light—just measure and results will improve.  

The challenge is that this catchy phrase doesn’t consider the complexity of human performance in organizations. We often measure to get a reaction, but we expect the workforce to react in predictable ways. In reality, measurement can have the intended, positive effect—increased motivation, aligned effort, and focus—or it can have unintended negative effects that result in counterproductive changes. 

Every Measure Motivates

It’s important to remember that every measure has the potential to motivate someone, somewhere. When you review marketing data to decide which product should receive more investment, someone is motivated to receive the investment. 

Consider balanced scorecards. Kaplan and Norton developed the scorecard as a strategic learning and steering tool. The scorecard is presented as a network of hypotheses that reflect strategy, referred to as a strategy map. The scorecard’s measures are intended to reveal whether the strategy and operations are working. Essentially, the measures test the validity of executive hypotheses and assumptions. 

As with many measurement systems, the intent of scorecards, at least initially, was to make decisions. Because each scorecard measure reflects directly on a different function within the organization, however, every measure will also motivate someone, somewhere in the organizational structure. In fact, a balanced scorecard is a motivational system. 

Regardless of incentives, balanced scorecard measures are scrutinized at the highest levels of an organization. Part of the motivation associated with scorecards is simply that people don’t want to look bad. 

Some organizations have gone so far as to link scorecard measures to individual executives. Simply reporting the measure makes the executive responsible. Other organizations have formalized this motivation system by developing cascading incentive systems that link the strategy of the organization to departments down and across the organization.

I’ve helped organizations use scorecards this way, and generally I see it as positive. It’s better to be deliberate in setting up motivation systems, because allowing random motivation systems to emerge can be damaging.  

Too often, organizations often proceed as if measures intended to be used for decision making won’t have a motivational effect. In other cases, organizations create measures with the intent to motivate, but assume that the measure will only have the intended effect. It’s important to remember, regardless of intent, that any measure will have motivational properties.

Strange Motivations

There are endless examples of the unintended motivational effects of measurement, also known as perverse incentives. One well known example is called the “Cobra Effect.” In British-controlled India, the government’s reward for every dead cobra—a reward intended to reduce the number of deadly snakes—resulted, of course, in people breeding cobras for income. A similar situation happened in Hanoi, under French colonial rule. In this case, the government paid a bounty for each rat pelt. Again, a program intended to exterminate rats instead led to rat farming.

But don’t think we’ve gotten smarter. IBM faced a similar problem when it decided to pay its programmers by the line. The programmers responded, predictably, by increasing the number of lines they wrote in each program. Instead of producing more programs, faster—the intended effect—the programmers simply wrote more complicated, and less elegant, code.

In some colleges, professors are rewarded for high scores on student evaluations. More often than not, this leads to easy courses and inflated grades, rather than improved accountability. Academic researchers are rewarded for a large number of publications. While the intent is to improve research productivity, the result is often incremental papers and little innovation. 

In the K–12 world, teachers are being rewarded for increased student test scores. Rather than improving education and teacher effectiveness, the effects have been teaching to the test and an emphasis on short-term learning.

Motivation and Measurement Are Personal and Social

The key to understanding these strange motivations is remembering that each employee, manager, and executive experiences his or her own context. Employees make meaning out of measures and measurement data, and they come to their own conclusions based on personal comparisons, context, and connotations. 

Personal conclusions and insights can be difficult to predict. All sorts of dysfunctional behavior can be observed in complex organizations. There are always challenges: What matters to one individual is often not what matters to the organization. It’s foolhardy to proceed under the assumption that all the people involved—measure developers, the executive team, employees—have the same worldview.

Many talent professionals assume that formal incentives, such as pay and advancement, are the primary motivators for employees. But there are other motivation systems have huge effects in organizations. These shadow incentives exert a powerful force on individuals. Personal relationships and social structures matter in organizations. This is the important conclusion of the Hawthorne studies, which showed that employee behavior, and organizational productivity,  is strongly related to social contexts.

A Shadow Incentive System

About a decade ago, a large telecommunications company started a program that encouraged repair technicians to develop their troubleshooting skills, in part by pursuing an associate’s degree in telecommunications technology. The company assumed that highly skilled technicians would be better at fixing problems, and that their increased skill would reduce the number of repeat service calls.

Despite the multi-million dollar price of the training initiative, repair technicians never repaired more than three phones a day. The company’s management, understandably baffled, hired a team of researchers to look into the problem.

The problem turned out to be shadow incentive system. An anthropologist joined the technicians in their trucks, watched their interactions, and found that the technicians had simply established a norm: three phone repairs per day. Anyone who worked faster was punished and shunned—serious disincentives. One technician who broke the rule had a tool dropped on his head by another worker on a pole above him.

This was the social element—team norms were strictly enforced. There was also a financial incentive. By reducing the number of repairs, the technicians were able to nearly double their income with overtime. The formal compensation and reward system simply didn’t matter. The shadow incentive system was much stronger. Ultimately, the program was discontinued.

What’s also interesting is that some workers weren’t consciously aware of the enforced repair limit. It took an anthropologist—an outsider—to see what was really happening. People often aren’t aware of the basis for their actions.

Understanding and building motivation systems requires insight on an individual and organizational level.

The Side Effect of Surveying Engagement

Many organizations now measure employee engagement with surveys, share the results, and hope the information will encourage managers and teams to improve. This feedback process is potentially positive and powerful.  

But conducting a survey—asking employees what’s wrong and how to make things better—can raise expectations. If the organization fails to make improvements based on survey feedback, the result can be the opposite of what was intended: lower morale. In these situations, survey results do little more than give dissatisfied employees something else to complain about.

In addition, managers and employees react to surveys and measurement according to their idiosyncratic worldviews. While one manager may work to improve engagement and expect her team to respond honestly, another manager may simply ask the team to rate survey questions higher, as a personal favor.

To manage the unintended side effects of surveys, we need to be aware of the expectations, strange motivations, and personal connotations that will inevitably come into play at different levels of the organization. 

If we’re aware of these different contexts, and the survey is framed in a forward-looking agenda, the results can help management focus employees on the positive aspects of improving the organization

In practice, it’s best to formally assign executives responsibility for the measures. Assigning accountability is going to happen anyway. Formalizing this effect increases transparency and openness across the organization.

Must We Measure Everything?

One of the challenges in measuring employees is that good measures are hard to find. If we’re going to evaluate a proofreader’s work, for example, the only way to measure the quality of the work would be to ask another proofreader. And who is going to evaluate the proofreader’s evaluation? Obviously, we can’t have a perfect measure of everything.

The lesson here is, don’t measure for the sake of measurement. A bad measure can create a bigger problem than not measuring at all.

If you can’t find a good measure, it might be worth looking for another way to monitor and motivate performance. In the case of the proofreader, you could consider surveying customers, who will have a sense of the quality of work. 

It’s worth asking the question: Do you really need a measure for this, or are you measuring because that’s what people do? 

Reducing Negative Side Effects: Managing Measures

As we have seen, measurement can lead to misalignment and malfunction in an organization or, for that matter, a country. Perhaps we should think about this differently: What gets measured needs to be managed. To be successful, we need to manage both the measure and its meaning.

If we can be deliberate in setting up motivation systems, being aware of the possibility of perverse incentives, it’s less likely that random motivations or shadow incentive systems will undermine the organization.

For measures to have their intended effect, it’s necessary to manage the meaning and the context. As always, communication is the key to successfully using measures.

I’ll write more about building shared meaning with measures in the next post.

Myths of Measurement: Perfect Employee Measures Exist

The full contribution of an employee can’t be measured with one metric. Over the last decade, I’ve asked thousands of HR professionals—including some very sophisticated ones—this question:

Can you give me an example of a measure that represents the total value an employee adds to the organization? 

No one has ever been able to do this. Sometimes they suggest a measure from a different company or a department that they don’t understand well. We want to think that somewhere, if we look hard enough, we’ll find the perfect measure of employee performance—it’s just that our own department or company is structured in such a way that we can’t do it.

Admittedly, this is a trick question. Everyone knows perfection doesn’t exist.

We want to think that our performance measurement system is really good—after all, we use it to make all sorts of decisions, including pay. On reflection, however, it’s clear that any measure of employee performance is deficient in that it doesn’t capture the entire value that the employee contributes. Any measure is contaminated by things beyond the employee’s control. Finally, any measure of an employee’s contribution is imprecise in that it isn’t really capable of making precise distinctions, either between people or between different aspects of a person. There is always an element of error, statistically speaking.

A good example is sales performance. Total sales may seem like a perfect measure until you consider the many factors that influence the measure. Any measure of total sales will be deficient because it doesn’t represent the good will associated with honestly representing a product. “Total sales” is short-term. Good will affects future sales. Every measure is deficient in that it doesn’t tell the whole story.

As a measure, “total sales” is also contaminated by factors beyond the sales representatives’ control—factors such as marketing, competition, and the quality or value of the product. Any seasoned sales representative will tell you that the territory will affect a salesperson’s ability to sell. In addition, all measures are imprecise. Consider two sales people: Both had a great year, and their annual sales were within a few thousand dollars of each other. Can we really differentiate between the skill, effort, or success of the two? No.

In practice, the myth of a perfect measure leads to misunderstandings about performance and capabilities. Here’s an example.

A call center manager was concerned about the employee taking the fewest calls. Clearly, this long-term employee was starting to slide. His performance had been in a slow decline.

The manager called the employee into his office. Being a sensitive boss, he asked the employee how things were going and if anything had changed in his life. Finally, he got down to business and confronted the employee about his performance—knowing there were no external factors affecting performance.

The employee’s response was unexpected: “I think I’m totally on top of my game. All my peers appreciate me, and I’m resolving the toughest issues. My productivity is very high!”

After a bit more discussion, the situation became clear. Informally, other call center representatives were escalating the difficult calls to the “problem” employee. If the employee is helping out a team member, he’s obviously contributing both experience and capability. This contribution, however, wouldn’t show up in his productivity measurement data. But his productivity was clearly high, considering that he was handling the most difficult calls.

In this case, the measure was deficient because it represented only quantity, not the quality or difficulty of the task. The employee’s performance was much broader than what was measured. Meanwhile, the other employees’ measurement data was contaminated—their productivity measures were a representation of the more experienced employee’s work. I would also bet this measure is imprecise—that it’s affected by random events, like the call-center computer going down.

This example shows that there can be great value in unmeasured aspects of performance, and that measurement can’t tell the whole story. In the best-case scenario, this call center manager would learn that it’s not possible to rely entirely on a measure that represents only one aspect of employee contribution.

 The Myth of Perfect Measurement in Action 

Organizations often act as if their measures are nearly perfect. As a consultant, I’m often asked to develop measures or provide measure-based guidance on important questions, such as “What value did we get from training?” It’s important to set expectations carefully. Without a perfect measure, there will be no perfect answer.

Early in my career, I spent months, and considerable amounts of a client’s money, searching for a good measure of technician performance for a regional phone company. In one sense, we had very good measures of contribution: the number of repairs completed successfully and the number of phone installations (this was in the era of copper lines and touch-tone phones).  But applying these measures in real-world context, even using strong quasi-experimental designs, we ran into many complications. In the process, my client and I learned, or relearned, a lot:

• Employees "game measures"
• Teamwork matters 
• Organizations are multilayered and complicated 
• Employee performance is multidimensional
• It’s difficult to isolate the contribution of an individual to the organization 
• It’s impossible to perfectly quantify the effect of an intervention (such as training) on an individual’s contribution. 

In the end, we were able to estimate the value of an employee training program. On the other hand, we didn’t use by-the-book analysis, and we certainly didn’t calculate a concrete ROI. Because measures are so imperfect, the ROI analysis would have been based on unrealistic assumptions. We were, however, able to make an educated projection. In the end, that was all that was needed to make good decisions about the training program.

Wouldn’t it be great if there were a perfect measure of employee performance? You could do all sorts of analyses and run organizations in a perfectly rational manner. Unfortunately, the perfect measure doesn’t exist. In fact, I’m not sure you can ever calculate the ROI of a training program, because there is never a perfect measure of employee performance—and that’s what training is supposed to affect.

Beyond this sort of program evaluation and decision making, many talent decisions are based on imperfect measures. As in the example of sales performance above, even sales commissions are based on measures that deficient, contaminated, and imprecise.

Consider More Subtle Outcomes

 In a general sense, phone technicians have clear outcomes: new phones are installed or repaired phones  work correctly. Many, in fact most jobs, have much less clear outcomes.

Consider a teacher in our public schools. Is the outcome of successful teach a grade on a test, success in life or something else.  Measuring the full contribution or value of these jobs is nearly impossible because the outcomes is so complicated. Even if we could define a clear outcome, its measure would be contaminated and deficient.  

What Can We Know? 

Even though all measures are deficient, contaminated, and imprecise, they can still be useful tools to support decision making, if we use them wisely. What’s critical is that we never assume that measures are perfect.

If we make this mistake—if we accept measures as having any degree of perfection—we shut down authentic conversations and prevent the exchange of meaning between managers, employees, and different parts of an organization. In the case of the call center described earlier, for example, relying on a measure without communication, and without the value of human judgment, could have resulted in the organization losing one of its most valuable employees.

Paradoxically, if we accept the inherent weakness of measurement, we can use measures more effectively. When we understand that measurement is an inherently simple approach that will never adequately describe the complexity of employees, or untangle the nuances of employee performance, we can compensate.

One way we can compensate is by relying on multiple measures—while accepting that even this solution is imperfect. Using multiple measures provides the benefit of triangulation, and is often adequate for making decisions, providing feedback, and even supporting human resource decisions.

Ultimately, the weaknesses of measurement provide an opportunity for discussion and learning. Authentic and productive communication that acknowledges these imperfections enhances learning about an organization and its complexities.

Measuring Invisible Talent

When we think about measurement, we usually think about measuring objects—their length or weight. The concrete nature of this sort of measurement is easy to understand and predictable: A meter is always a meter.

Talent measurement, in contrast, is not easy to understand, and it’s unpredictable. Measuring talent has its challenges, but it’s one of the keys to organizational learning and employee motivation.

When it comes to measuring talent, we look for consistent and concrete measures, just as we do in the physical world. We hope for honesty in the mathematical precision offered by measures and metrics.  We hope for less wiggle room and more candor. We hope measurement data provides less theory, better insights, and obvious decisions. 

Comparing physical and talent helps us to see the value and potential of measurement. The value is high, but if we cling to the metaphor of physical measurement, we will grow frustrated. We may also miss one of the real strengths of talent measurement.

Talent measurement is different and it is complicated. Among the complications, it has a special attribute: measurement motivates. To access the benefits of talent measurement we need to consider how physical and talent measures differ. 

Talent Is Invisible

Talent measures aren’t concrete, like their physical counterparts. Many of the most important assets of today’s world are essentially invisible—think of wealth, power, relationships, personality, or intelligence.  Because the aspects of talent that we care about are invisible, it can be difficult to know what we are measuring.    

It’s not just that talent is invisible. We also need to remember that measures are just a representation of talent. This adds complications. We all know that someone’s height in inches isn’t the person. It’s easy, however, to confuse a measure of potential with the value of a high-potential employee. The measure of potential is a representation of an underlying capability, and the measure is accurate only in a probabilistic sense.   

Invisibility and representation are two reasons that talent measures tend to be less precise, and less consistent, than physical measures. Have two managers rate how well an employee completed a difficult task, and you’re likely to get two different answers. Ask two skilled carpenters to measure the length of a cabinet, and you’ll probably get two nearly identical answers, accurate to within one-sixteenth of an inch. 

Despite this, organizations often act as though their measures are nearly perfect. For example, some management consultants recommend that employees be ranked annually. Let’s be clear about what ranking actually means: Employees will be listed in order, from best to worst. To truly rank employees, there would need to be a distinction between the 10^th and 11^th best employees. Without a perfect measure, this is impossible. Since we don’t have measures that are up to this task, we would have to use other means to rank employees, such as intuition. 

Ranking employees by a physical measure sounds easier, but even this is complicated. Let’s say you’re at a family gathering and you’re taking a photo of your grandparents, siblings, nieces, nephews, and so forth. You’d ask people to organize themselves by height—shorter people in front—so that the camera can capture their faces. We often think about employees this way: We can just line them up according to some feature, such as performance. We’ll keep the best, or the tallest, and get rid of the rest.

Of course, it’s not that simple! Even physical measurement is imperfect. Imagine trying to get 1,000 employees to stand in order, by height. I can hear the questions already. Does big hair count? Should we take our shoes off? In the end, we’d probably need to ask ourselves, "what is employee height, anyway?"

If we take physical measurement to this logical conclusion, it provides a useful lesson: measurement is more complicated in practice than in theory. We may think we understand what leadership is. When it comes to measuring it, we need to get pretty specific in our meaning.  

People React to Being Measured

In general, physical measurement has few side effects. If you measure the length of a cabinet, you don’t affect the cabinet, and the cabinet isn’t likely to react. Talent, unlike inanimate objects, is affected in complicated ways by measurement. People react to measures.

In fairness, some physical objects are affected by measurement. When checking tire pressure, a small amount of air escapes. This affects the tire pressure. This is a simple example of an observer effect, which has been well documented in physics. For example, a glass thermometer absorbs thermal energy when taking a measurement. 

Observer effects on physical objects are generally unsurprising and small. Talent’s reaction to measurement is complicated and can be large. 

The effect can be positive. Measurement can lead to motivation, increased effort, and more focus.  Feedback and reasonable goals often lead to higher levels of performance, as we discussed in past blog posts.  

Practical experience however, shows that this isn’t always the case. Unfortunately, measurement can change talent in counterproductive ways, depending on the context. Measurement can de-motivate and distract. If measures are linked to very difficult goals, employees sometimes give up, or—even worse—lose faith in the organization and disengage. 

To make things worse, reactions to measurement can also motivate talent to corrupt or game the measures. Physical measurement never has this issue. Humans, however, have a major preoccupation with gaming measures. It’s so common that the famous methodologist Don Campbell, discussing program evaluation in 1975, described what has become known as Campbell's Law:

The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.   

Given these complexities, perhaps talent management should steal the nickname “dismal science” from economics:

• The underlying dimension you wish to understand is invisible 
• Measuring talent is imprecise and probabilistic 
•  Humans react to measures, sometimes by changing themselves, and sometimes by changing the measure.

 

So, Do We Stop Measuring Talent?

It’s been said that since we can’t measure talent perfectly, we should simply give up and acknowledge defeat. For 30 years, some consultants have advocated getting rid of performance appraisals altogether. I’m amazed this impractical idea is still being considered.

If we don’t measure human performance, we lose a powerful motivational and learning tool. As with many aspects in life, we simply have to manage the dilemma and tension. 

Measurement is flawed and we must use it.  We can’t hide our head in the sand and hope these flaws will go away. The flaws are inherent in measuring, especially something as complicated as talent. If we step into the real world of human idiosyncrasies, measurement is a powerful tool that can help us improve organizational performance, ensure educational excellence, and motivate personal growth. 

In the coming blog posts I will further elaborate the myths of talent measurement and how we can think more clearly for organizational learning, motivation and growth.

Charley Morrow