Myths of Measurement: Talent Measures Have No Side Effects

Medication and medical procedures have side effects. Side effects can be harmless (too much niacin can cause blushing) or serious (statins can cause muscle damage). These medications work within the complex chemistry of the human body, and how a person will experience a side effect is unpredictable.

Talent measures are not that different. They work within complex human social systems, and they alter these systems—often in unintended ways. In other words, they have side effects. The side effects most often affect motivation.

There are many ways to increase organizational performance. Many are effective, few are ineffective. And a few can actually damage the organization. Measurement falls into this last category. Because measurement can have negative outcomes, it is worth knowing how these results come about and how they can be managed

When We Measure, We Expect Results

Talent measurement often has the goal of understanding, motivating, or directing workforce performance.  This idea is summarized in the phrase what gets measured gets managed, which is sometimes attributed to Peter Drucker. This phrase is universally framed in a positive light—just measure and results will improve.  

The challenge is that this catchy phrase doesn’t consider the complexity of human performance in organizations. We often measure to get a reaction, but we expect the workforce to react in predictable ways. In reality, measurement can have the intended, positive effect—increased motivation, aligned effort, and focus—or it can have unintended negative effects that result in counterproductive changes. 

Every Measure Motivates

It’s important to remember that every measure has the potential to motivate someone, somewhere. When you review marketing data to decide which product should receive more investment, someone is motivated to receive the investment. 

Consider balanced scorecards. Kaplan and Norton developed the scorecard as a strategic learning and steering tool. The scorecard is presented as a network of hypotheses that reflect strategy, referred to as a strategy map. The scorecard’s measures are intended to reveal whether the strategy and operations are working. Essentially, the measures test the validity of executive hypotheses and assumptions. 

As with many measurement systems, the intent of scorecards, at least initially, was to make decisions. Because each scorecard measure reflects directly on a different function within the organization, however, every measure will also motivate someone, somewhere in the organizational structure. In fact, a balanced scorecard is a motivational system. 

Regardless of incentives, balanced scorecard measures are scrutinized at the highest levels of an organization. Part of the motivation associated with scorecards is simply that people don’t want to look bad. 

Some organizations have gone so far as to link scorecard measures to individual executives. Simply reporting the measure makes the executive responsible. Other organizations have formalized this motivation system by developing cascading incentive systems that link the strategy of the organization to departments down and across the organization.

I’ve helped organizations use scorecards this way, and generally I see it as positive. It’s better to be deliberate in setting up motivation systems, because allowing random motivation systems to emerge can be damaging.  

Too often, organizations often proceed as if measures intended to be used for decision making won’t have a motivational effect. In other cases, organizations create measures with the intent to motivate, but assume that the measure will only have the intended effect. It’s important to remember, regardless of intent, that any measure will have motivational properties.

Strange Motivations

There are endless examples of the unintended motivational effects of measurement, also known as perverse incentives. One well known example is called the “Cobra Effect.” In British-controlled India, the government’s reward for every dead cobra—a reward intended to reduce the number of deadly snakes—resulted, of course, in people breeding cobras for income. A similar situation happened in Hanoi, under French colonial rule. In this case, the government paid a bounty for each rat pelt. Again, a program intended to exterminate rats instead led to rat farming.

But don’t think we’ve gotten smarter. IBM faced a similar problem when it decided to pay its programmers by the line. The programmers responded, predictably, by increasing the number of lines they wrote in each program. Instead of producing more programs, faster—the intended effect—the programmers simply wrote more complicated, and less elegant, code.

In some colleges, professors are rewarded for high scores on student evaluations. More often than not, this leads to easy courses and inflated grades, rather than improved accountability. Academic researchers are rewarded for a large number of publications. While the intent is to improve research productivity, the result is often incremental papers and little innovation. 

In the K–12 world, teachers are being rewarded for increased student test scores. Rather than improving education and teacher effectiveness, the effects have been teaching to the test and an emphasis on short-term learning.

Motivation and Measurement Are Personal and Social

The key to understanding these strange motivations is remembering that each employee, manager, and executive experiences his or her own context. Employees make meaning out of measures and measurement data, and they come to their own conclusions based on personal comparisons, context, and connotations. 

Personal conclusions and insights can be difficult to predict. All sorts of dysfunctional behavior can be observed in complex organizations. There are always challenges: What matters to one individual is often not what matters to the organization. It’s foolhardy to proceed under the assumption that all the people involved—measure developers, the executive team, employees—have the same worldview.

Many talent professionals assume that formal incentives, such as pay and advancement, are the primary motivators for employees. But there are other motivation systems have huge effects in organizations. These shadow incentives exert a powerful force on individuals. Personal relationships and social structures matter in organizations. This is the important conclusion of the Hawthorne studies, which showed that employee behavior, and organizational productivity,  is strongly related to social contexts.

A Shadow Incentive System

About a decade ago, a large telecommunications company started a program that encouraged repair technicians to develop their troubleshooting skills, in part by pursuing an associate’s degree in telecommunications technology. The company assumed that highly skilled technicians would be better at fixing problems, and that their increased skill would reduce the number of repeat service calls.

Despite the multi-million dollar price of the training initiative, repair technicians never repaired more than three phones a day. The company’s management, understandably baffled, hired a team of researchers to look into the problem.

The problem turned out to be shadow incentive system. An anthropologist joined the technicians in their trucks, watched their interactions, and found that the technicians had simply established a norm: three phone repairs per day. Anyone who worked faster was punished and shunned—serious disincentives. One technician who broke the rule had a tool dropped on his head by another worker on a pole above him.

This was the social element—team norms were strictly enforced. There was also a financial incentive. By reducing the number of repairs, the technicians were able to nearly double their income with overtime. The formal compensation and reward system simply didn’t matter. The shadow incentive system was much stronger. Ultimately, the program was discontinued.

What’s also interesting is that some workers weren’t consciously aware of the enforced repair limit. It took an anthropologist—an outsider—to see what was really happening. People often aren’t aware of the basis for their actions.

Understanding and building motivation systems requires insight on an individual and organizational level.

The Side Effect of Surveying Engagement

Many organizations now measure employee engagement with surveys, share the results, and hope the information will encourage managers and teams to improve. This feedback process is potentially positive and powerful.  

But conducting a survey—asking employees what’s wrong and how to make things better—can raise expectations. If the organization fails to make improvements based on survey feedback, the result can be the opposite of what was intended: lower morale. In these situations, survey results do little more than give dissatisfied employees something else to complain about.

In addition, managers and employees react to surveys and measurement according to their idiosyncratic worldviews. While one manager may work to improve engagement and expect her team to respond honestly, another manager may simply ask the team to rate survey questions higher, as a personal favor.

To manage the unintended side effects of surveys, we need to be aware of the expectations, strange motivations, and personal connotations that will inevitably come into play at different levels of the organization. 

If we’re aware of these different contexts, and the survey is framed in a forward-looking agenda, the results can help management focus employees on the positive aspects of improving the organization

In practice, it’s best to formally assign executives responsibility for the measures. Assigning accountability is going to happen anyway. Formalizing this effect increases transparency and openness across the organization.

Must We Measure Everything?

One of the challenges in measuring employees is that good measures are hard to find. If we’re going to evaluate a proofreader’s work, for example, the only way to measure the quality of the work would be to ask another proofreader. And who is going to evaluate the proofreader’s evaluation? Obviously, we can’t have a perfect measure of everything.

The lesson here is, don’t measure for the sake of measurement. A bad measure can create a bigger problem than not measuring at all.

If you can’t find a good measure, it might be worth looking for another way to monitor and motivate performance. In the case of the proofreader, you could consider surveying customers, who will have a sense of the quality of work. 

It’s worth asking the question: Do you really need a measure for this, or are you measuring because that’s what people do? 

Reducing Negative Side Effects: Managing Measures

As we have seen, measurement can lead to misalignment and malfunction in an organization or, for that matter, a country. Perhaps we should think about this differently: What gets measured needs to be managed. To be successful, we need to manage both the measure and its meaning.

If we can be deliberate in setting up motivation systems, being aware of the possibility of perverse incentives, it’s less likely that random motivations or shadow incentive systems will undermine the organization.

For measures to have their intended effect, it’s necessary to manage the meaning and the context. As always, communication is the key to successfully using measures.

I’ll write more about building shared meaning with measures in the next post.

Motivating with Measures: Accountability, Incentives and the Dark Side

The benefits and risks of using measures for motivation are amplified when employees are made accountable or incentivized.

Measurement and Accountability

Measurement is at the heart of accountability. In the dictionary, accountability has a neutral meaning: an obligation or willingness to accept responsibility for one’s actions. This is the denotative, or literal, meaning. In a work setting, the denotative meaning of accountability is a goal that defines who will do what by when.

Accountability in this sense is the basis of management by objectives  (MBO).  While MBOs were popularized in the 1950s, they remain a central element of most organizations’ annual performance appraisals. 

While some objectives are task-based, the best objectives are measurement-based. We have found that the most effective examples of accountability-based motivation use SMART goals—goals that are Specific, Measurable, Agreed upon, Realistic, and Time-bound. Describing expectations in terms of measures at the beginning of a project motivates performance. 

Accountability is a word often loaded with connotations. How would you feel if you were told in a business meeting that you will be held accountable? Queasy? The phrase suggests that you’re in trouble. This isn’t actually accountability—it’s scapegoating. This is a connotative meaning, and the connotations of accountability are negative. The fear of negative consequences can lead to all sorts of dysfunctional behavior.

If the meaning of the measures isn’t managed, then accountability is more likely to instill a culture of fear than it is to motivate employees and support the organization’s strategic goals. 

To motivate with measurement-based accountability, the meaning of the measure must be managed. Use measures to describe expectations before the employee works to achieve results.  Articulate both the formal, denotative meaning (how the measure works) and the connotative meaning (the implications for the employee).

• If you’re building a measurement system, remember that the connotations are probably more important than the measures. Consider how the measures will be seen by employees. Develop a list of actions employees could take to influence the data.  Be sure to include actions that the system intends to encourage as well as unintended actions. Adapt your system accordingly to especially encourage the intended and discourage the unintended.
• If you’re managing employee accountability with measures, be sure to talk with employees about both the denotative and connotative meanings. It’s important to develop a shared vision: This is a true leadership communication task.  If there is clear agreement on the meaning of the measure as well as the level of performance expected, accountability can be positive. 

Accountability is simply responsibility. Measurement can help top build responsibility for results and the rewards or consequences of the results.

Incentives and Measurement

If goal setting and accountability work, why not add incentives to make them work even better? Why not juice the motivation system? Many of us have worked, or currently work, in an incentive system. Sales people work on commission. Managers get bonuses and stock options.

There is a whole industry of compensation consultants trying to create incentives that work. Since the industrial revolution, we’ve been trying to get incentives right—and some of us are starting to wonder if incentives are just wrong.  

Research summarized by Daniel Pink suggests that incentives lead to lower performance in completing tasks that are complex or involve creative thinking. I’m sure we’ll find that this relationship is true in many circumstances. 

I am also sure there are many circumstances in which incentives lead to improved performance, even in complex and creative tasks. As with many aspects of human performance, there are complexities.

Life isn’t one-size-fits-all; there are individual differences and nuances of context that influence how incentives affect performance. Long-term goals, which are difficult to study experimentally, may work better with incentives.  Mr Pink presents the world in black and white; i am confident there are many shades of gray.

There is a bigger problem with linking incentives to measures, however.  Incentives, or consequences, have the tendency to put the focus exclusively on moving the needle—on affecting the data and the measure rather than addressing the underlying goal. Too much focus on the connotations of the measure, as opposed to the meaning of the measure, leads to gaming.

The Dilbert comic strip may seem ridiculous, but as is always the case in Scott Adams’ cartoons, absurdity reflects reality to an uncomfortable degree (many of his cartoons are based on real-life examples submitted by readers). Incentives can have unintended consequences, often encouraging employees to behave unethically. For example, if you were earning a subsistence wage as a packer for Green Giant, and the company announced that a bonus would be paid to every employee who could find and remove insect parts from packages of frozen peas, what would you do? Possibly what many of the employees did—bring insect parts from home to earn the incentive.

There are, of course, more troubling examples of the dark side of measurement-based motivation. In the sad story of system-wide cheating in Atlanta Public Schools, 178 employees, including both teachers and principals, are now suspected of inflating scores on standardized tests to earn the significant rewards that come with rapid improvements in school performance. Outright swindles, such as Bernie Madoff, are all too common.

In sales departments there are more subtle examples of gaming incentive systems. Sales departments have been known to count all sales in the current quarter toward commissions—even though many of the sales are not actually closed. 

Conclusion

It’s dangerous to rely too much on measures for motivation: The more you emphasize measures, the more apt the measures are to cause dysfunctional, even unethical, behavior.  If you need to use measures for accountability and incentives, be careful.  Measures can’t replace management; they are a management tool.  It is necessary to make sure that the measures are reasonable – not gamed – and that accountability is understood and positive.