Myths of Measurement: Talent Measures Have No Side Effects

Medication and medical procedures have side effects. Side effects can be harmless (too much niacin can cause blushing) or serious (statins can cause muscle damage). These medications work within the complex chemistry of the human body, and how a person will experience a side effect is unpredictable.

Talent measures are not that different. They work within complex human social systems, and they alter these systems—often in unintended ways. In other words, they have side effects. The side effects most often affect motivation.

There are many ways to increase organizational performance. Many are effective, few are ineffective. And a few can actually damage the organization. Measurement falls into this last category. Because measurement can have negative outcomes, it is worth knowing how these results come about and how they can be managed

When We Measure, We Expect Results

Talent measurement often has the goal of understanding, motivating, or directing workforce performance.  This idea is summarized in the phrase what gets measured gets managed, which is sometimes attributed to Peter Drucker. This phrase is universally framed in a positive light—just measure and results will improve.  

The challenge is that this catchy phrase doesn’t consider the complexity of human performance in organizations. We often measure to get a reaction, but we expect the workforce to react in predictable ways. In reality, measurement can have the intended, positive effect—increased motivation, aligned effort, and focus—or it can have unintended negative effects that result in counterproductive changes. 

Every Measure Motivates

It’s important to remember that every measure has the potential to motivate someone, somewhere. When you review marketing data to decide which product should receive more investment, someone is motivated to receive the investment. 

Consider balanced scorecards. Kaplan and Norton developed the scorecard as a strategic learning and steering tool. The scorecard is presented as a network of hypotheses that reflect strategy, referred to as a strategy map. The scorecard’s measures are intended to reveal whether the strategy and operations are working. Essentially, the measures test the validity of executive hypotheses and assumptions. 

As with many measurement systems, the intent of scorecards, at least initially, was to make decisions. Because each scorecard measure reflects directly on a different function within the organization, however, every measure will also motivate someone, somewhere in the organizational structure. In fact, a balanced scorecard is a motivational system. 

Regardless of incentives, balanced scorecard measures are scrutinized at the highest levels of an organization. Part of the motivation associated with scorecards is simply that people don’t want to look bad. 

Some organizations have gone so far as to link scorecard measures to individual executives. Simply reporting the measure makes the executive responsible. Other organizations have formalized this motivation system by developing cascading incentive systems that link the strategy of the organization to departments down and across the organization.

I’ve helped organizations use scorecards this way, and generally I see it as positive. It’s better to be deliberate in setting up motivation systems, because allowing random motivation systems to emerge can be damaging.  

Too often, organizations often proceed as if measures intended to be used for decision making won’t have a motivational effect. In other cases, organizations create measures with the intent to motivate, but assume that the measure will only have the intended effect. It’s important to remember, regardless of intent, that any measure will have motivational properties.

Strange Motivations

There are endless examples of the unintended motivational effects of measurement, also known as perverse incentives. One well known example is called the “Cobra Effect.” In British-controlled India, the government’s reward for every dead cobra—a reward intended to reduce the number of deadly snakes—resulted, of course, in people breeding cobras for income. A similar situation happened in Hanoi, under French colonial rule. In this case, the government paid a bounty for each rat pelt. Again, a program intended to exterminate rats instead led to rat farming.

But don’t think we’ve gotten smarter. IBM faced a similar problem when it decided to pay its programmers by the line. The programmers responded, predictably, by increasing the number of lines they wrote in each program. Instead of producing more programs, faster—the intended effect—the programmers simply wrote more complicated, and less elegant, code.

In some colleges, professors are rewarded for high scores on student evaluations. More often than not, this leads to easy courses and inflated grades, rather than improved accountability. Academic researchers are rewarded for a large number of publications. While the intent is to improve research productivity, the result is often incremental papers and little innovation. 

In the K–12 world, teachers are being rewarded for increased student test scores. Rather than improving education and teacher effectiveness, the effects have been teaching to the test and an emphasis on short-term learning.

Motivation and Measurement Are Personal and Social

The key to understanding these strange motivations is remembering that each employee, manager, and executive experiences his or her own context. Employees make meaning out of measures and measurement data, and they come to their own conclusions based on personal comparisons, context, and connotations. 

Personal conclusions and insights can be difficult to predict. All sorts of dysfunctional behavior can be observed in complex organizations. There are always challenges: What matters to one individual is often not what matters to the organization. It’s foolhardy to proceed under the assumption that all the people involved—measure developers, the executive team, employees—have the same worldview.

Many talent professionals assume that formal incentives, such as pay and advancement, are the primary motivators for employees. But there are other motivation systems have huge effects in organizations. These shadow incentives exert a powerful force on individuals. Personal relationships and social structures matter in organizations. This is the important conclusion of the Hawthorne studies, which showed that employee behavior, and organizational productivity,  is strongly related to social contexts.

A Shadow Incentive System

About a decade ago, a large telecommunications company started a program that encouraged repair technicians to develop their troubleshooting skills, in part by pursuing an associate’s degree in telecommunications technology. The company assumed that highly skilled technicians would be better at fixing problems, and that their increased skill would reduce the number of repeat service calls.

Despite the multi-million dollar price of the training initiative, repair technicians never repaired more than three phones a day. The company’s management, understandably baffled, hired a team of researchers to look into the problem.

The problem turned out to be shadow incentive system. An anthropologist joined the technicians in their trucks, watched their interactions, and found that the technicians had simply established a norm: three phone repairs per day. Anyone who worked faster was punished and shunned—serious disincentives. One technician who broke the rule had a tool dropped on his head by another worker on a pole above him.

This was the social element—team norms were strictly enforced. There was also a financial incentive. By reducing the number of repairs, the technicians were able to nearly double their income with overtime. The formal compensation and reward system simply didn’t matter. The shadow incentive system was much stronger. Ultimately, the program was discontinued.

What’s also interesting is that some workers weren’t consciously aware of the enforced repair limit. It took an anthropologist—an outsider—to see what was really happening. People often aren’t aware of the basis for their actions.

Understanding and building motivation systems requires insight on an individual and organizational level.

The Side Effect of Surveying Engagement

Many organizations now measure employee engagement with surveys, share the results, and hope the information will encourage managers and teams to improve. This feedback process is potentially positive and powerful.  

But conducting a survey—asking employees what’s wrong and how to make things better—can raise expectations. If the organization fails to make improvements based on survey feedback, the result can be the opposite of what was intended: lower morale. In these situations, survey results do little more than give dissatisfied employees something else to complain about.

In addition, managers and employees react to surveys and measurement according to their idiosyncratic worldviews. While one manager may work to improve engagement and expect her team to respond honestly, another manager may simply ask the team to rate survey questions higher, as a personal favor.

To manage the unintended side effects of surveys, we need to be aware of the expectations, strange motivations, and personal connotations that will inevitably come into play at different levels of the organization. 

If we’re aware of these different contexts, and the survey is framed in a forward-looking agenda, the results can help management focus employees on the positive aspects of improving the organization

In practice, it’s best to formally assign executives responsibility for the measures. Assigning accountability is going to happen anyway. Formalizing this effect increases transparency and openness across the organization.

Must We Measure Everything?

One of the challenges in measuring employees is that good measures are hard to find. If we’re going to evaluate a proofreader’s work, for example, the only way to measure the quality of the work would be to ask another proofreader. And who is going to evaluate the proofreader’s evaluation? Obviously, we can’t have a perfect measure of everything.

The lesson here is, don’t measure for the sake of measurement. A bad measure can create a bigger problem than not measuring at all.

If you can’t find a good measure, it might be worth looking for another way to monitor and motivate performance. In the case of the proofreader, you could consider surveying customers, who will have a sense of the quality of work. 

It’s worth asking the question: Do you really need a measure for this, or are you measuring because that’s what people do? 

Reducing Negative Side Effects: Managing Measures

As we have seen, measurement can lead to misalignment and malfunction in an organization or, for that matter, a country. Perhaps we should think about this differently: What gets measured needs to be managed. To be successful, we need to manage both the measure and its meaning.

If we can be deliberate in setting up motivation systems, being aware of the possibility of perverse incentives, it’s less likely that random motivations or shadow incentive systems will undermine the organization.

For measures to have their intended effect, it’s necessary to manage the meaning and the context. As always, communication is the key to successfully using measures.

I’ll write more about building shared meaning with measures in the next post.

Myths of Measurement: Talent Measures Are Unaffected by Context

Physical measurement is barely affected by context: an inch is always an inch, and a cabinet always has the same dimensions, whether you are building it, improving it, or removing it.

This is not true of talent measures; talent measures are extremely sensitive to context. The same measure will yield different results in different contexts—whether you are selecting, developing, or laying off employees. It’s not a good idea to assume that you can use one talent measure for different purposes.

To understand the prevalent myth that talent measures are unaffected by context, we need to understand that measurement is just a method of conveying information: that is, measurement is a language. While the mathematical language of measurement is more precise than spoken language, meaning will vary with context.

Consider a competency rating. If the measure is used to set compensation many will only see the measure as a gateway to pay. If the organization uses the measure for two different purposes—compensation and developmental coaching—the coaching context will be contaminated by the context of pay. When it comes to competency ratings, employees often pay more attention to the context than to the measure itself.

Context is often more important than the measure. Let’s look at a few examples.

The Context of Performance Management

Most organizations have an annual performance appraisal. In most cases, an organization will review the number of ratings at each point on the scale (the distribution). Given obvious variability in performance, we would expect a normal distribution—a few employees would receive high ratings, and a few low ratings, but the great majority would cluster around the middle of the scale. In most organizations, however, nearly all the employees are clustered at the top of the scale, and only a few fall near the bottom. The distribution is skewed.

Over the years, skewed performance ratings have caused consternation, difficult conversations, and organizational chaos. Executives have looked at the distribution of performance ratings and thought:

• We sure have a great workforce—everybody is doing well!
• This measure is obviously biased—I know our workforce is not that great.

It should come as no surprise that the ratings are skewed, considering the context. Because the ratings may affect compensation, bosses tend to give higher ratings. The social context, not the measurement process, is causing the skew. Nevertheless, many organizations look for a better measure to provide more differentiation between employees, or to increase the number of low-rated employees. 

No matter how many times you change the performance appraisal measure, you’re unlikely to get a different distribution. The context stays the same, and as a result, the distribution is likely to stay the same. The employee/boss relationship will lead to a preponderance of positive ratings, and changing the measurement system will never solve the problem of skewed performance appraisals.

Solutions such as forced ranking, which I have discussed elsewhere as inappropriate, are simply masked attempts to develop a better measure. They may change the distribution, but they suffer from other problems such as spurious differences.

In any organization, the solution to skewed ratings in performance appraisal won’t be a better measure. Certainly, it is easy to change the measure.  Further, there are many different tweaks that can be made, including different rating scales, number of points on the rating scale, different dimensions to rate.  If, however, you really want to change the distribution, develop better management discipline and use the existing measurement system. This requires discipline difficult conversations between bosses and the employees who work for them. 

The Context of Employee Engagement

There is currently a small revolution happening in employers’ views of employee engagement. Starting with Marcus Buckingham’s research linking engagement survey results to positive outcomes such as productivity, customer satisfaction, and employee retention, employers have rediscovered employee surveys. Many executives worry about an unengaged workforce and the impact on their business, and many employers are surveying their workforce for the first time.

Some organizations have even linked incentives to engagement measures—for example, by increasing or decreasing a manager’s compensation based on the engagement scores in his or her area. This practice seems justified, since we can find relationships between engagement and outcomes such as profitability and retention. Anything that can be done to increase engagement should be tried.

As with performance appraisals, however, adding financial incentives will fundamentally change the context of the measure. Employees have told me, in confidence, that their manager asked them to respond to the survey positively, regardless of how they were feeling. One of the most disengaging things a manager can do is to ask an employee to misrepresent herself. The effect will be an extreme form of contamination of the measure. While actual engagement will decrease, the measure will show an increase. This is a form of cheating.

Another Name for the Myth

Psychologists who work with performance measures have developed a term for how measures are changed by context: When a measure of performance is affected by non-performance factors, they refer to it as criterion contamination. Because the performance variable will be contaminated by the context, researchers are warned not to use performance appraisal results when conducting research. If they do, the research will not yield meaningful results.

The various social and motivational forces that affect performance appraisals are one example of context. There are many other examples of contextual influence: organizational culture, business processes, personal beliefs, discipline, and so forth. These contexts affect every type of talent measures.

Organizations often forget about criterion contamination and try to use a single measure for different purposes. If a measure has been linked to incentive pay, for example, it’s not possible to use the same measure to study the relationship between employee engagement and customer satisfaction. The measure has been contaminated by the compensation context, and the context is always more powerful than the measure.

In this case, the solution to criterion contamination is to get a new measure. It’s certainly inconvenient to develop additional measures, especially when a perfectly good measure already exists.

The Same Context May Be More Different Than You Think

As the engagement example above shows, just as the meaning of measures varies according to the organizational context, it also varies according to individual context. This is another aspect of the myth of unaffected measures: there is often an assumption that the measure means the same thing to you and me.

As discussed in previous posts, this is the connotative, or subjective, meaning of the measure. Although in a denotative sense the measure will have exactly the same meaning at any level of an organization, within that organization, the measures will mean radically different things to different groups and different individuals.

In the engagement-gaming example above, for example, the context of the measure varies between the different parties:

• Executive management is concerned with engagement and its impact on the business in terms of productivity, customer satisfaction, or employee engagement 
• Supervisors have incentive compensation and are concerned with how the scores will affect their pay 
• Employees feel pressure to respond positively, but may have insights to share—once again the system is preventing them from having a voice.

Recognizing the existence, and the effect, of connotative meanings presents one of the biggest challenges in talent measurement. If we pay attention to the connotative meanings—that is, the individual and group contexts surrounding a measure—we can communicate to create shared meaning. In a culture of open communication, there is a significant opportunity to get more value from measures.

To Use Measures Well, Remember the Myth

Organizations need employees who are engaged in achieving organizational goals. This idea goes by many names, such as ownership culture and results orientation. Measures are often used to encourage engagement, with the intent of building a shared worldview and an understanding of the organization.  Performance appraisals and scorecards help keep everyone on the same page. Or do they?

It’s important to remember how easily measures are changed—some would say corrupted—by context. Misuse a measure once and employees will remember it for a long time. Use a performance appraisal for laying off employees, and this will change the context in the future. It’s easy for a measure to pick up new connotations.

Human resource departments and leaders have an opportunity to manage the meaning of talent measures at all levels of an organization. One way to do this is to watch for this myth in action. Remembering that every talent measure is affected by context can lead to a more discerning use of measurement, better communication, and, ultimately, more positive outcomes.

Of course, I’m not the first to point out this challenge. In 1975, Donald T. Campbell observed a methodological phenomenon that some refer to as Campbell’s Law: 

The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

In the nearly 40 years since Campbell’s observation, talent measures and performance metrics have proliferated in organizations. The proliferation is accelerating as measurement becomes inexpensive and accessible. In my experience, however, few consider this dark side of measurement.

In the next blog post, I’ll consider how talent, which Campbell refers to as “the underlying social process,” is affected by measurement.