Myths of Measurement: Talent Measures Have No Side Effects

Medication and medical procedures have side effects. Side effects can be harmless (too much niacin can cause blushing) or serious (statins can cause muscle damage). These medications work within the complex chemistry of the human body, and how a person will experience a side effect is unpredictable.

Talent measures are not that different. They work within complex human social systems, and they alter these systems—often in unintended ways. In other words, they have side effects. The side effects most often affect motivation.

There are many ways to increase organizational performance. Many are effective, few are ineffective. And a few can actually damage the organization. Measurement falls into this last category. Because measurement can have negative outcomes, it is worth knowing how these results come about and how they can be managed

When We Measure, We Expect Results

Talent measurement often has the goal of understanding, motivating, or directing workforce performance.  This idea is summarized in the phrase what gets measured gets managed, which is sometimes attributed to Peter Drucker. This phrase is universally framed in a positive light—just measure and results will improve.  

The challenge is that this catchy phrase doesn’t consider the complexity of human performance in organizations. We often measure to get a reaction, but we expect the workforce to react in predictable ways. In reality, measurement can have the intended, positive effect—increased motivation, aligned effort, and focus—or it can have unintended negative effects that result in counterproductive changes. 

Every Measure Motivates

It’s important to remember that every measure has the potential to motivate someone, somewhere. When you review marketing data to decide which product should receive more investment, someone is motivated to receive the investment. 

Consider balanced scorecards. Kaplan and Norton developed the scorecard as a strategic learning and steering tool. The scorecard is presented as a network of hypotheses that reflect strategy, referred to as a strategy map. The scorecard’s measures are intended to reveal whether the strategy and operations are working. Essentially, the measures test the validity of executive hypotheses and assumptions. 

As with many measurement systems, the intent of scorecards, at least initially, was to make decisions. Because each scorecard measure reflects directly on a different function within the organization, however, every measure will also motivate someone, somewhere in the organizational structure. In fact, a balanced scorecard is a motivational system. 

Regardless of incentives, balanced scorecard measures are scrutinized at the highest levels of an organization. Part of the motivation associated with scorecards is simply that people don’t want to look bad. 

Some organizations have gone so far as to link scorecard measures to individual executives. Simply reporting the measure makes the executive responsible. Other organizations have formalized this motivation system by developing cascading incentive systems that link the strategy of the organization to departments down and across the organization.

I’ve helped organizations use scorecards this way, and generally I see it as positive. It’s better to be deliberate in setting up motivation systems, because allowing random motivation systems to emerge can be damaging.  

Too often, organizations often proceed as if measures intended to be used for decision making won’t have a motivational effect. In other cases, organizations create measures with the intent to motivate, but assume that the measure will only have the intended effect. It’s important to remember, regardless of intent, that any measure will have motivational properties.

Strange Motivations

There are endless examples of the unintended motivational effects of measurement, also known as perverse incentives. One well known example is called the “Cobra Effect.” In British-controlled India, the government’s reward for every dead cobra—a reward intended to reduce the number of deadly snakes—resulted, of course, in people breeding cobras for income. A similar situation happened in Hanoi, under French colonial rule. In this case, the government paid a bounty for each rat pelt. Again, a program intended to exterminate rats instead led to rat farming.

But don’t think we’ve gotten smarter. IBM faced a similar problem when it decided to pay its programmers by the line. The programmers responded, predictably, by increasing the number of lines they wrote in each program. Instead of producing more programs, faster—the intended effect—the programmers simply wrote more complicated, and less elegant, code.

In some colleges, professors are rewarded for high scores on student evaluations. More often than not, this leads to easy courses and inflated grades, rather than improved accountability. Academic researchers are rewarded for a large number of publications. While the intent is to improve research productivity, the result is often incremental papers and little innovation. 

In the K–12 world, teachers are being rewarded for increased student test scores. Rather than improving education and teacher effectiveness, the effects have been teaching to the test and an emphasis on short-term learning.

Motivation and Measurement Are Personal and Social

The key to understanding these strange motivations is remembering that each employee, manager, and executive experiences his or her own context. Employees make meaning out of measures and measurement data, and they come to their own conclusions based on personal comparisons, context, and connotations. 

Personal conclusions and insights can be difficult to predict. All sorts of dysfunctional behavior can be observed in complex organizations. There are always challenges: What matters to one individual is often not what matters to the organization. It’s foolhardy to proceed under the assumption that all the people involved—measure developers, the executive team, employees—have the same worldview.

Many talent professionals assume that formal incentives, such as pay and advancement, are the primary motivators for employees. But there are other motivation systems have huge effects in organizations. These shadow incentives exert a powerful force on individuals. Personal relationships and social structures matter in organizations. This is the important conclusion of the Hawthorne studies, which showed that employee behavior, and organizational productivity,  is strongly related to social contexts.

A Shadow Incentive System

About a decade ago, a large telecommunications company started a program that encouraged repair technicians to develop their troubleshooting skills, in part by pursuing an associate’s degree in telecommunications technology. The company assumed that highly skilled technicians would be better at fixing problems, and that their increased skill would reduce the number of repeat service calls.

Despite the multi-million dollar price of the training initiative, repair technicians never repaired more than three phones a day. The company’s management, understandably baffled, hired a team of researchers to look into the problem.

The problem turned out to be shadow incentive system. An anthropologist joined the technicians in their trucks, watched their interactions, and found that the technicians had simply established a norm: three phone repairs per day. Anyone who worked faster was punished and shunned—serious disincentives. One technician who broke the rule had a tool dropped on his head by another worker on a pole above him.

This was the social element—team norms were strictly enforced. There was also a financial incentive. By reducing the number of repairs, the technicians were able to nearly double their income with overtime. The formal compensation and reward system simply didn’t matter. The shadow incentive system was much stronger. Ultimately, the program was discontinued.

What’s also interesting is that some workers weren’t consciously aware of the enforced repair limit. It took an anthropologist—an outsider—to see what was really happening. People often aren’t aware of the basis for their actions.

Understanding and building motivation systems requires insight on an individual and organizational level.

The Side Effect of Surveying Engagement

Many organizations now measure employee engagement with surveys, share the results, and hope the information will encourage managers and teams to improve. This feedback process is potentially positive and powerful.  

But conducting a survey—asking employees what’s wrong and how to make things better—can raise expectations. If the organization fails to make improvements based on survey feedback, the result can be the opposite of what was intended: lower morale. In these situations, survey results do little more than give dissatisfied employees something else to complain about.

In addition, managers and employees react to surveys and measurement according to their idiosyncratic worldviews. While one manager may work to improve engagement and expect her team to respond honestly, another manager may simply ask the team to rate survey questions higher, as a personal favor.

To manage the unintended side effects of surveys, we need to be aware of the expectations, strange motivations, and personal connotations that will inevitably come into play at different levels of the organization. 

If we’re aware of these different contexts, and the survey is framed in a forward-looking agenda, the results can help management focus employees on the positive aspects of improving the organization

In practice, it’s best to formally assign executives responsibility for the measures. Assigning accountability is going to happen anyway. Formalizing this effect increases transparency and openness across the organization.

Must We Measure Everything?

One of the challenges in measuring employees is that good measures are hard to find. If we’re going to evaluate a proofreader’s work, for example, the only way to measure the quality of the work would be to ask another proofreader. And who is going to evaluate the proofreader’s evaluation? Obviously, we can’t have a perfect measure of everything.

The lesson here is, don’t measure for the sake of measurement. A bad measure can create a bigger problem than not measuring at all.

If you can’t find a good measure, it might be worth looking for another way to monitor and motivate performance. In the case of the proofreader, you could consider surveying customers, who will have a sense of the quality of work. 

It’s worth asking the question: Do you really need a measure for this, or are you measuring because that’s what people do? 

Reducing Negative Side Effects: Managing Measures

As we have seen, measurement can lead to misalignment and malfunction in an organization or, for that matter, a country. Perhaps we should think about this differently: What gets measured needs to be managed. To be successful, we need to manage both the measure and its meaning.

If we can be deliberate in setting up motivation systems, being aware of the possibility of perverse incentives, it’s less likely that random motivations or shadow incentive systems will undermine the organization.

For measures to have their intended effect, it’s necessary to manage the meaning and the context. As always, communication is the key to successfully using measures.

I’ll write more about building shared meaning with measures in the next post.

Myths of Measurement: Perfect Employee Measures Exist

The full contribution of an employee can’t be measured with one metric. Over the last decade, I’ve asked thousands of HR professionals—including some very sophisticated ones—this question:

Can you give me an example of a measure that represents the total value an employee adds to the organization? 

No one has ever been able to do this. Sometimes they suggest a measure from a different company or a department that they don’t understand well. We want to think that somewhere, if we look hard enough, we’ll find the perfect measure of employee performance—it’s just that our own department or company is structured in such a way that we can’t do it.

Admittedly, this is a trick question. Everyone knows perfection doesn’t exist.

We want to think that our performance measurement system is really good—after all, we use it to make all sorts of decisions, including pay. On reflection, however, it’s clear that any measure of employee performance is deficient in that it doesn’t capture the entire value that the employee contributes. Any measure is contaminated by things beyond the employee’s control. Finally, any measure of an employee’s contribution is imprecise in that it isn’t really capable of making precise distinctions, either between people or between different aspects of a person. There is always an element of error, statistically speaking.

A good example is sales performance. Total sales may seem like a perfect measure until you consider the many factors that influence the measure. Any measure of total sales will be deficient because it doesn’t represent the good will associated with honestly representing a product. “Total sales” is short-term. Good will affects future sales. Every measure is deficient in that it doesn’t tell the whole story.

As a measure, “total sales” is also contaminated by factors beyond the sales representatives’ control—factors such as marketing, competition, and the quality or value of the product. Any seasoned sales representative will tell you that the territory will affect a salesperson’s ability to sell. In addition, all measures are imprecise. Consider two sales people: Both had a great year, and their annual sales were within a few thousand dollars of each other. Can we really differentiate between the skill, effort, or success of the two? No.

In practice, the myth of a perfect measure leads to misunderstandings about performance and capabilities. Here’s an example.

A call center manager was concerned about the employee taking the fewest calls. Clearly, this long-term employee was starting to slide. His performance had been in a slow decline.

The manager called the employee into his office. Being a sensitive boss, he asked the employee how things were going and if anything had changed in his life. Finally, he got down to business and confronted the employee about his performance—knowing there were no external factors affecting performance.

The employee’s response was unexpected: “I think I’m totally on top of my game. All my peers appreciate me, and I’m resolving the toughest issues. My productivity is very high!”

After a bit more discussion, the situation became clear. Informally, other call center representatives were escalating the difficult calls to the “problem” employee. If the employee is helping out a team member, he’s obviously contributing both experience and capability. This contribution, however, wouldn’t show up in his productivity measurement data. But his productivity was clearly high, considering that he was handling the most difficult calls.

In this case, the measure was deficient because it represented only quantity, not the quality or difficulty of the task. The employee’s performance was much broader than what was measured. Meanwhile, the other employees’ measurement data was contaminated—their productivity measures were a representation of the more experienced employee’s work. I would also bet this measure is imprecise—that it’s affected by random events, like the call-center computer going down.

This example shows that there can be great value in unmeasured aspects of performance, and that measurement can’t tell the whole story. In the best-case scenario, this call center manager would learn that it’s not possible to rely entirely on a measure that represents only one aspect of employee contribution.

 The Myth of Perfect Measurement in Action 

Organizations often act as if their measures are nearly perfect. As a consultant, I’m often asked to develop measures or provide measure-based guidance on important questions, such as “What value did we get from training?” It’s important to set expectations carefully. Without a perfect measure, there will be no perfect answer.

Early in my career, I spent months, and considerable amounts of a client’s money, searching for a good measure of technician performance for a regional phone company. In one sense, we had very good measures of contribution: the number of repairs completed successfully and the number of phone installations (this was in the era of copper lines and touch-tone phones).  But applying these measures in real-world context, even using strong quasi-experimental designs, we ran into many complications. In the process, my client and I learned, or relearned, a lot:

• Employees "game measures"
• Teamwork matters 
• Organizations are multilayered and complicated 
• Employee performance is multidimensional
• It’s difficult to isolate the contribution of an individual to the organization 
• It’s impossible to perfectly quantify the effect of an intervention (such as training) on an individual’s contribution. 

In the end, we were able to estimate the value of an employee training program. On the other hand, we didn’t use by-the-book analysis, and we certainly didn’t calculate a concrete ROI. Because measures are so imperfect, the ROI analysis would have been based on unrealistic assumptions. We were, however, able to make an educated projection. In the end, that was all that was needed to make good decisions about the training program.

Wouldn’t it be great if there were a perfect measure of employee performance? You could do all sorts of analyses and run organizations in a perfectly rational manner. Unfortunately, the perfect measure doesn’t exist. In fact, I’m not sure you can ever calculate the ROI of a training program, because there is never a perfect measure of employee performance—and that’s what training is supposed to affect.

Beyond this sort of program evaluation and decision making, many talent decisions are based on imperfect measures. As in the example of sales performance above, even sales commissions are based on measures that deficient, contaminated, and imprecise.

Consider More Subtle Outcomes

 In a general sense, phone technicians have clear outcomes: new phones are installed or repaired phones  work correctly. Many, in fact most jobs, have much less clear outcomes.

Consider a teacher in our public schools. Is the outcome of successful teach a grade on a test, success in life or something else.  Measuring the full contribution or value of these jobs is nearly impossible because the outcomes is so complicated. Even if we could define a clear outcome, its measure would be contaminated and deficient.  

What Can We Know? 

Even though all measures are deficient, contaminated, and imprecise, they can still be useful tools to support decision making, if we use them wisely. What’s critical is that we never assume that measures are perfect.

If we make this mistake—if we accept measures as having any degree of perfection—we shut down authentic conversations and prevent the exchange of meaning between managers, employees, and different parts of an organization. In the case of the call center described earlier, for example, relying on a measure without communication, and without the value of human judgment, could have resulted in the organization losing one of its most valuable employees.

Paradoxically, if we accept the inherent weakness of measurement, we can use measures more effectively. When we understand that measurement is an inherently simple approach that will never adequately describe the complexity of employees, or untangle the nuances of employee performance, we can compensate.

One way we can compensate is by relying on multiple measures—while accepting that even this solution is imperfect. Using multiple measures provides the benefit of triangulation, and is often adequate for making decisions, providing feedback, and even supporting human resource decisions.

Ultimately, the weaknesses of measurement provide an opportunity for discussion and learning. Authentic and productive communication that acknowledges these imperfections enhances learning about an organization and its complexities.

Talent Measurement Schools of Thought

Here is a puzzle: In our day-to-day life we do not treat people as inanimate objects—but we try to measure them as if they are! We treat the people in front of us as living, breathing, reacting entities, but few consider the complexity and reactivity of human nature when developing or managing with measures. Why the inconsistency?

Two Schools of Thought: The Taylor and Mayo Dichotomy

To solve this puzzle, you have to go back to school—graduate school. As a graduate student, you’ll probably learn one of two different approaches to talent measurement. One school of thought is focused on the technical aspect of measurement, and the other on the human aspect. The challenge for measurement professionals is to master both schools of thought. The two are rarely reconciled, however. Professionals generally have expertise primarily in one approach.

The technical or engineering school will teach you how to calculate reliability and validity, and introduce you to different measurement methods. This school of thought dates back to Frederick Taylor, one of the first manufacturing engineers. Frederick Taylor is considered the father of scientific management, which emphasizes task analysis, efficiency studies, time-and-motion studies, and using compensation schemes for motivation. 

The human relations school has a different point of view: Employees are complicated, and don’t work mechanistically. If your graduate program emphasizes human relations, you’re likely learn more about personality types or team functioning measures that will facilitate interactions between people at work. You’ll be introduced to validity and reliability, but you’ll be taught very little about the technology and theory of measures. The human relations school of thought dates back to Elton Mayo, a psychologist. 

The ghosts of Taylor and Mayo haunt today’s organizations. To this day, consultants, managers, and leaders adhere to one school or the other. Taylor adherents tend to advocate for measurement as a formal and rigid process. Mayo adherents focus more on group processes, interpersonal communication, and intrinsic motivation. 

Both Taylor and Mayo made essential contributions to the art of management and leadership. But it’s not an either/or choice. It often takes decades of experience to merge the two schools of thought into a practical working knowledge of measurement. Some never see the dichotomy and its implications.

I’m writing this blog post in the hope that we can accelerate the process of combining and ultimately uniting these two schools of measurement.

The Engineer: Frederick Winslow Taylor (1856 – 1915) “In the past the man has been first; in the future the system must be first.”

Taylor grew up affluent and gifted in the second half of the 19^th century, in an era of huge industrial change. He chose not to follow his father into the legal profession, although he was accepted into Harvard. Instead, he worked in industry, starting as a machinist and becoming a foreman, and went on to study engineering. 

As an engineer, he first improved manufacturing technology such as lathes and forging equipment. Early on, he noticed that these technical improvements demanded similar organizational innovations to be effective. As his ideas developed, he saw manufacturing as a larger system that could be improved by optimizing the various pieces to contribute to the larger system. Over the course of his career, he contributed his ideas to equipment (he had several important patents), business processes (such as accounting methods), and methods of managing employees.  

Taylor and Time-and-Motion Studies

As he looked at the larger manufacturing picture, Taylor was concerned that laborers were not working at full capacity. To fix this problem, he identified the optimum work-output level, and provided incentive pay for this level of output.  

Determining workers’ optimum output involved time-and-motion studies. Taylor divided the work into steps, each of which he timed separately. He then combined the time for each step into a total time for the job. By dividing the work day by the total job time, he arrived at an optimum production rate.

Workers were paid on a graduated scale. Low levels of output were paid very little, but as productivity approached the maximum, unit pay increased. Workers attaining the optimum production rate would be paid 60% more using Taylor’s methods. 

While he became infamous for his time-and-motion studies, it’s important to recognize that, for Taylor, these studies were part of a larger system of managing employees. Taylor used worker productivity as a talent measure. He studied measures of productivity to make decisions, organize work, set production expectations, motivate employees, and identify employees to retain. In the best cases, Taylor’s scientific management methods could reduce costs and increase productivity by 50% to 100%.

Human reaction to measures and management methods didn’t factor into Taylor’s thinking. He was convinced that employees only work for money. Labor problems were simply an engineering challenge to be managed. Taylor paid lip service to selecting and developing talent—he mostly set output targets. Workers who were able to keep up the pace self-selected and developed their capability.

Taylor’s blind spot—the human factor—can be seen in many contemporary organizational improvement interventions, such as re-engineering, which has a success rate as low as 30%. Human readiness and acceptance of change is often a barrier to re-engineering success.   

Taylor’s approach also was inconsistent. Sometimes it worked, sometimes it led to significant problems.

Employee reactions to Taylor’s intervention often led to work actions and strikes.  Ultimately there was an congressional investigation. By the time of Taylor’s death at age 59, Congress had outlawed use of stopwatches and bonus payments in the federal government. Scientific management was increasingly discredited.

The Humanist: George Elton Mayo (1880 – 1949) “So long as … business methods take no account of human nature … expect strikes and sabotage to be the ordinary.”

Mayo grew up in a distinguished Australian family. He began his studies in medicine and ended up studying psychology, focusing on social interactions at work. His most famous research work can be found in the Hawthorne studies, which demonstrated that employees are largely influenced by social factors, and that they react to being observed.

Mayo’s most important work coincided with the Great Depression. He believed that the industrial revolution had shattered strong social relationships in the workplace, and he found that workers acted according to sentiments and emotion. He felt that if managers treated workers with respect and tried to meet their needs, then both workers and management would benefit.  

Mayo’s research indicated that belonging to a group is a more powerful motivator than money. In his management philosophy, he saw attitudes, proper supervision, and informal social relationships as the key to productivity.

Some consider Mayo’s work to be a reaction to Taylorism. But Mayo was also concerned with output and productivity. Unlike Taylor, however, he was interested in the social and psychological interventions that increased productivity. These interventions are indeed helpful, and understanding the human factor is critical.  

Thanks to Mayo’s work, we recognize that, in organizations, informal social structures matter as much as formal structures, such as the chain of command. For example, a likeable senior engineer who dislikes a new manager could undermine the manager’s authority by making jokes at his expense during every meeting. In effect, the engineer becomes more influential than the manager—outside the hierarchy of the organizational chart.

Today, many organizational interventions emphasize team-building, and are based on the recognition that organizational culture is important, and managers have ongoing relationships with employees. By acknowledging the importance of the informal structure of an organization, factors such as relationships, informal leadership, and influence can be aligned with organizational needs and direction.

Mayo’s insights were synthesized into a school of thought referred to as human relations. The human relations school continues strong to this day, often in the form of leadership development, team building, or change initiatives. 

The insight missed by Mayo is that measurement—even Taylor’s productivity measurements—are essentially a social process. Measurement is simply a method of communication—a way to make meaning between groups.

Since Mayo, many people have failed to make this essential connection: We can extend Mayo’s insight into the importance of informal (social) structures into an understanding of the importance of the informal (connotative or personal) meanings of measures. As I have discussed before, the informal meanings of measures matter as much as, if not more than, their formal meanings. Like social structures, these connotative meanings can be managed—but only when their existence and importance are acknowledged.

If you’re creating an organizational, and you follow Taylor, you may believe that compensation is the sole motivation for performance and advancement. If you follow Mayo, you may believe that love, fear, and other ineffable human factors are the primary motivators. 

In the same way, the designers of a formal measurement system may believe that their measures will motivate by providing people with a positive opportunity to make more money (the denotative meaning). Instead, the designers may find that the connotative meanings provoke reactions that ultimately trump their intentions—reactions from outright rejection to gaming the system.

It’s odd that many of our measurement systems haven’t progressed beyond Taylor’s way of thinking. We have many tools to address the informal meanings of measures—tools we can draw on from interpersonal communications theory, management practices, and organizational learning.

Another danger of following Mayo’s approach is that it often pays too much attention to the informal and emergent social structures of an organization. While these informal structures are powerful influences on individual performance, it is possible to merge formal and informal organizational structures into a shared structure. This is where measurement can be incredibly effective, if it is used as a means of communication: It can create shared meaning that bridges the organizational and personal definitions of performance, motivation, and reward.

Finding a Middle Ground

What’s most unfortunate about the Mayo vs. Taylor bifurcation is that they were both right: The difference between the two schools of thought is ideological, not practical. In practice, we use both approaches. We need both engineers and social scientists (psychologist and sociologists) to run organizations efficiently.

If we attend one graduate school, we may learn to develop measures that are technically good, but we’ll have trouble assessing the human reaction to measurement. If we attend another school, we may learn to facilitate social interaction and meaning, but we won’t be trained to motivate, direct, or improve performance through measurement and feedback. Personally, I attended a more technical school, but my life and work experiences have led me to appreciate a balanced approach.

What Taylor missed was the importance of social structures in motivation, and the human factor in reaction to measurement. What Mayo missed was that measurement in itself is a social process, and measures have informal (social) meanings that can be managed.

Today, 80 years after Mayo’s Hawthorne studies, we should be able to merge the two schools. There is a wealth of possibilities for applying Taylor’s ideas in measuring individual productivity. At the same time, we’ve vastly increased our understanding of human relations—there’s a huge industry that’s evolved out of Mayo’s original insights.

Finally, in resolving the polarity of these two approaches, we need to acknowledge that measurement is communication, and that communication is shared meaning. By starting with a simple point—that people always react to measurement, and that the reaction is unpredictable—we can take the denotative and connotative meanings of measures, the formal and informal structures in organizations, and the two schools of thought, and synthesize them into an elegant, effective approach to talent measurement.  

Myths of Measurement: Do Measures Reflect Reality?

In the last blog post we discussed the mental models that inform our understanding of talent. Today’s post will examine how measures make mental models explicit and useful. This is true of talent management and other fields. 

I’d also like to discuss how easy it is to misunderstand talent measures as concrete entities. Just as there was a danger in reifying our mental models of talent, it’s easy to forget that measurement results are just a numerical representation of a model. The model is not “real,” and the measures, for all their predictive or descriptive strength, are just a representation of the model.

Mental and Mathematical Models

When measuring talent, we develop mathematical models to represent our mental models.  Often we start with a conceptual model, which is a sketchy idea. An operational model, on the other hand, is precisely specified in mathematical language. Operational models often have good predictive or descriptive strength. 

This is similar to an architect’s process. An architect starts a project by drawing a conceptual sketch, and refines the sketch into a scale plan. Sometimes it turns out that the original ideas don’t work. Sometimes the scale plan makes the concepts more workable. Scaling the concept mathematically makes it more predictive, more descriptive, and more useful.  

Refining Measures

Operational measures and scales are strong tools, and often work well to summarize personality, results, potential, or competency. The numerical values of the scales can be compared and linked to other values such as compensation. They can also be tested.

As an architect may find that her concept won’t work in practice, we may find that a talent measure does not work as we conceptualized it. For example, if we compare measures of performance and personality to investigate our mental model that extroverts are better at sales, we may find that personality does not relate to performance as we expected.

Statistics can help us refine and strengthen our talent measures. If we find that an employee engagement survey is only weakly related to customer satisfaction, we can add survey questions to strengthen the relationship. Adding questions about the organizational climate, such as “my co-workers really care about the customer’s experience,” is likely to increase the correlation. Examining statistical correlations can help us develop a measure that’s quite important to the business.  

Personality assessments are among the most refined talent measures. Many personality instruments have been revised over the years—the state of the art, in some cases, is astounding. The Hogan Personality Inventory (HPI), which defines personality as social reputation, has now undergone 30 years of refinements. It was developed by correlating respondents’ answers to survey questions with friends’ and co-workers’ descriptions of the respondents (social reputation). Today, the 206 questions of the survey—questions such as “I would like to be a race-car driver”—allow surprisingly accurate assessment and precise differentiation between different aspects of personality. 

Many assessment participants feel that the HPI can read their minds, but the “wow” factor is simply produced by probabilistic relationships between survey questions and reputation. In a sense, it’s the magic of statistics—“any sufficiently advanced technology is indistinguishable from magic” (Arthur C. Clarke). However, participants’ feelings that the HPI personality instrument can see their true selves can easily lead to reification.

Of course, not all personality instruments are as well refined as the HPI, and it’s important to remember that even the HPI is probabilistic. These instruments are accurate nearly all the time, but not always. Imperfections are easy to overlook because the instruments are “right” so often, and in general. Overlooking the imperfections, however, has dangers.

How Reification Happens

There is something about putting numbers on a model that makes the model seem real and unquestionable. But this presents a problem. When we can’t ask questions about our models, we can’t learn.  

For some reason, it’s easy to accept mathematical talent measurement results as the truth, and not look beyond the numbers. I have some theories about why this reification happens.

• Some people aren’t as comfortable with numbers as they are with words. If it’s a lot of work for an individual to understand a chart or a report full of numbers, it’s likely that the person will only review the measures superficially. It’s also less likely that the person will ask questions. 
•  The basis of talent measures isn’t always made clear. When providing HPI feedback, we don’t explain conceptually or computationally how the scales were developed or scored. In fact, the calculation methods are a secret known only to the Hogans. In one sense, it’s not important to know these details. But in another sense, not understanding how a measure works—or having no access to the mechanism behind the measures—could lead to reification. 
•  When the talent measures are rigidly used for decision making, for example compensation or selection, the are in a sense real. Certainly they control real outcomes.  

 

Reification and the History of Intelligence Testing

The danger of measure reification is obvious in the long and often sad history of intelligence testing. In 1905, Alfred Binet proposed  a method to measure intelligence in children. A careful scientist, he noted the method’s limitations: 

This scale properly speaking does not permit the measure of … intelligence, because intellectual qualities … cannot be measured as linear surfaces are measured.

Binet intended to develop a tool to classify children needing attention. He tried to not reify the underlying capability.

Since then, intelligence has been reified and recast as a real and invariable human attribute—an attribute that describes a limit of human potential. The application of intelligence testing has limited access to immigration, schools, and jobs.  

When we reify a measure, we extend the measure beyond its original design. In this case, research indicates that intelligence does change. In addition, capabilities such as emotional intelligence are more important for some jobs. Making decisions based solely on employee intelligence is a mistake.  Intelligence quotient is not a real thing. It is a measure developed for a specific and narrow task: identifying children who need attention to succeed academically. Use in industry, and for immigration, came much later.

While many would argue with me, I assert that intelligence must be combined with other measures to be useful in business.

Reification and the Danger of Self-Fulfilling Prophecies

Reifying measures can lead to self-fulfilling prophecies. For example, designating an employee as “high potential” one year often means they will continue to be seen as high potential in future years, regardless of changes in performance. This is similar to calling a student “gifted.”

When a manager gives a low performance rating to an employee, there can be similar long-term consequences. People often conform to expectations. This is called the Pygmalion effect, which is well studied in schools. The Pygmalion effect also happens in organizations. 

Reification and the Danger of Limited Thinking

Unquestioning acceptance of any representative model is a problem because it limits our ability to think broadly about a situation. We tend to think that a talent measure describes talent completely. If we do this, we fall into the trap of mistaking the map for the territory.

Early sea charts were representations of mariners’ mental models. They were crude but adequate for coastal navigation at the time. Today they seem wildly imaginative and mostly decorative. But partly as a result of the maps’ reification of these mental models, sailors stayed close to shore to avoid the monsters, whirlpools, and other dangers that became very real to them—including the danger of sailing over the edge of the world.  

Sometimes, we stay close to what is familiar. If we’re familiar with the idea of intelligence, we refer to someone as smart. If we’re familiar with descriptions of personality, we may refer to a person as an introvert.  But there is much more to a person than our mental models, and our measures, would suggest.

Recognizing the Limits of Measures Is the Key to Using Them Well

Ultimately, talent measures are just representations of mental models. The underlying talent is always much more complicated. Any representation, or model, is necessarily a simplification.

I am concerned that we take measures as better, and more, than they actually are. If we don’t consider the limits of the tools, the limits of the tools become our limits.

I don’t think we should look for more perfect measures of talent. I am certain they do not exist. For one thing, the available technology reflects our current understanding of talent. 

So, throwing out our current talent measures is probably not helpful. Instead, we can do better by increasing our understanding of the current measures. This is an evolutionary process, and a probably a process that must be done in collaboration with others. How else can we examine our assumptions, and question both our measures and the underlying mental models on which they’re based? (I’ll be talking extensively about building shared meaning of measures in future blogs)

If we’re to use our measures intelligently, we won’t expect them to be more perfect than they are—even if they’re mathematically correct 95% of the time. We’ll remember that measures are never true representations of reality: A measure can never contain the whole truth, the total complexity of a person, or an entire situation. And we won’t allow ourselves to be daunted by the “truth” of numerical measures, which leads us to accept them superficially. Instead, we can use measures as a starting point for thoughtful exploration and deeper communication. 

It’s important to remember that all measures represent someone’s theory. The theory may not be appropriate in the current context, and may not be measured well

Measuring Invisible Talent

When we think about measurement, we usually think about measuring objects—their length or weight. The concrete nature of this sort of measurement is easy to understand and predictable: A meter is always a meter.

Talent measurement, in contrast, is not easy to understand, and it’s unpredictable. Measuring talent has its challenges, but it’s one of the keys to organizational learning and employee motivation.

When it comes to measuring talent, we look for consistent and concrete measures, just as we do in the physical world. We hope for honesty in the mathematical precision offered by measures and metrics.  We hope for less wiggle room and more candor. We hope measurement data provides less theory, better insights, and obvious decisions. 

Comparing physical and talent helps us to see the value and potential of measurement. The value is high, but if we cling to the metaphor of physical measurement, we will grow frustrated. We may also miss one of the real strengths of talent measurement.

Talent measurement is different and it is complicated. Among the complications, it has a special attribute: measurement motivates. To access the benefits of talent measurement we need to consider how physical and talent measures differ. 

Talent Is Invisible

Talent measures aren’t concrete, like their physical counterparts. Many of the most important assets of today’s world are essentially invisible—think of wealth, power, relationships, personality, or intelligence.  Because the aspects of talent that we care about are invisible, it can be difficult to know what we are measuring.    

It’s not just that talent is invisible. We also need to remember that measures are just a representation of talent. This adds complications. We all know that someone’s height in inches isn’t the person. It’s easy, however, to confuse a measure of potential with the value of a high-potential employee. The measure of potential is a representation of an underlying capability, and the measure is accurate only in a probabilistic sense.   

Invisibility and representation are two reasons that talent measures tend to be less precise, and less consistent, than physical measures. Have two managers rate how well an employee completed a difficult task, and you’re likely to get two different answers. Ask two skilled carpenters to measure the length of a cabinet, and you’ll probably get two nearly identical answers, accurate to within one-sixteenth of an inch. 

Despite this, organizations often act as though their measures are nearly perfect. For example, some management consultants recommend that employees be ranked annually. Let’s be clear about what ranking actually means: Employees will be listed in order, from best to worst. To truly rank employees, there would need to be a distinction between the 10^th and 11^th best employees. Without a perfect measure, this is impossible. Since we don’t have measures that are up to this task, we would have to use other means to rank employees, such as intuition. 

Ranking employees by a physical measure sounds easier, but even this is complicated. Let’s say you’re at a family gathering and you’re taking a photo of your grandparents, siblings, nieces, nephews, and so forth. You’d ask people to organize themselves by height—shorter people in front—so that the camera can capture their faces. We often think about employees this way: We can just line them up according to some feature, such as performance. We’ll keep the best, or the tallest, and get rid of the rest.

Of course, it’s not that simple! Even physical measurement is imperfect. Imagine trying to get 1,000 employees to stand in order, by height. I can hear the questions already. Does big hair count? Should we take our shoes off? In the end, we’d probably need to ask ourselves, "what is employee height, anyway?"

If we take physical measurement to this logical conclusion, it provides a useful lesson: measurement is more complicated in practice than in theory. We may think we understand what leadership is. When it comes to measuring it, we need to get pretty specific in our meaning.  

People React to Being Measured

In general, physical measurement has few side effects. If you measure the length of a cabinet, you don’t affect the cabinet, and the cabinet isn’t likely to react. Talent, unlike inanimate objects, is affected in complicated ways by measurement. People react to measures.

In fairness, some physical objects are affected by measurement. When checking tire pressure, a small amount of air escapes. This affects the tire pressure. This is a simple example of an observer effect, which has been well documented in physics. For example, a glass thermometer absorbs thermal energy when taking a measurement. 

Observer effects on physical objects are generally unsurprising and small. Talent’s reaction to measurement is complicated and can be large. 

The effect can be positive. Measurement can lead to motivation, increased effort, and more focus.  Feedback and reasonable goals often lead to higher levels of performance, as we discussed in past blog posts.  

Practical experience however, shows that this isn’t always the case. Unfortunately, measurement can change talent in counterproductive ways, depending on the context. Measurement can de-motivate and distract. If measures are linked to very difficult goals, employees sometimes give up, or—even worse—lose faith in the organization and disengage. 

To make things worse, reactions to measurement can also motivate talent to corrupt or game the measures. Physical measurement never has this issue. Humans, however, have a major preoccupation with gaming measures. It’s so common that the famous methodologist Don Campbell, discussing program evaluation in 1975, described what has become known as Campbell's Law:

The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.   

Given these complexities, perhaps talent management should steal the nickname “dismal science” from economics:

• The underlying dimension you wish to understand is invisible 
• Measuring talent is imprecise and probabilistic 
•  Humans react to measures, sometimes by changing themselves, and sometimes by changing the measure.

 

So, Do We Stop Measuring Talent?

It’s been said that since we can’t measure talent perfectly, we should simply give up and acknowledge defeat. For 30 years, some consultants have advocated getting rid of performance appraisals altogether. I’m amazed this impractical idea is still being considered.

If we don’t measure human performance, we lose a powerful motivational and learning tool. As with many aspects in life, we simply have to manage the dilemma and tension. 

Measurement is flawed and we must use it.  We can’t hide our head in the sand and hope these flaws will go away. The flaws are inherent in measuring, especially something as complicated as talent. If we step into the real world of human idiosyncrasies, measurement is a powerful tool that can help us improve organizational performance, ensure educational excellence, and motivate personal growth. 

In the coming blog posts I will further elaborate the myths of talent measurement and how we can think more clearly for organizational learning, motivation and growth.

Charley Morrow

Motivating with Measures: Accountability, Incentives and the Dark Side

The benefits and risks of using measures for motivation are amplified when employees are made accountable or incentivized.

Measurement and Accountability

Measurement is at the heart of accountability. In the dictionary, accountability has a neutral meaning: an obligation or willingness to accept responsibility for one’s actions. This is the denotative, or literal, meaning. In a work setting, the denotative meaning of accountability is a goal that defines who will do what by when.

Accountability in this sense is the basis of management by objectives  (MBO).  While MBOs were popularized in the 1950s, they remain a central element of most organizations’ annual performance appraisals. 

While some objectives are task-based, the best objectives are measurement-based. We have found that the most effective examples of accountability-based motivation use SMART goals—goals that are Specific, Measurable, Agreed upon, Realistic, and Time-bound. Describing expectations in terms of measures at the beginning of a project motivates performance. 

Accountability is a word often loaded with connotations. How would you feel if you were told in a business meeting that you will be held accountable? Queasy? The phrase suggests that you’re in trouble. This isn’t actually accountability—it’s scapegoating. This is a connotative meaning, and the connotations of accountability are negative. The fear of negative consequences can lead to all sorts of dysfunctional behavior.

If the meaning of the measures isn’t managed, then accountability is more likely to instill a culture of fear than it is to motivate employees and support the organization’s strategic goals. 

To motivate with measurement-based accountability, the meaning of the measure must be managed. Use measures to describe expectations before the employee works to achieve results.  Articulate both the formal, denotative meaning (how the measure works) and the connotative meaning (the implications for the employee).

• If you’re building a measurement system, remember that the connotations are probably more important than the measures. Consider how the measures will be seen by employees. Develop a list of actions employees could take to influence the data.  Be sure to include actions that the system intends to encourage as well as unintended actions. Adapt your system accordingly to especially encourage the intended and discourage the unintended.
• If you’re managing employee accountability with measures, be sure to talk with employees about both the denotative and connotative meanings. It’s important to develop a shared vision: This is a true leadership communication task.  If there is clear agreement on the meaning of the measure as well as the level of performance expected, accountability can be positive. 

Accountability is simply responsibility. Measurement can help top build responsibility for results and the rewards or consequences of the results.

Incentives and Measurement

If goal setting and accountability work, why not add incentives to make them work even better? Why not juice the motivation system? Many of us have worked, or currently work, in an incentive system. Sales people work on commission. Managers get bonuses and stock options.

There is a whole industry of compensation consultants trying to create incentives that work. Since the industrial revolution, we’ve been trying to get incentives right—and some of us are starting to wonder if incentives are just wrong.  

Research summarized by Daniel Pink suggests that incentives lead to lower performance in completing tasks that are complex or involve creative thinking. I’m sure we’ll find that this relationship is true in many circumstances. 

I am also sure there are many circumstances in which incentives lead to improved performance, even in complex and creative tasks. As with many aspects of human performance, there are complexities.

Life isn’t one-size-fits-all; there are individual differences and nuances of context that influence how incentives affect performance. Long-term goals, which are difficult to study experimentally, may work better with incentives.  Mr Pink presents the world in black and white; i am confident there are many shades of gray.

There is a bigger problem with linking incentives to measures, however.  Incentives, or consequences, have the tendency to put the focus exclusively on moving the needle—on affecting the data and the measure rather than addressing the underlying goal. Too much focus on the connotations of the measure, as opposed to the meaning of the measure, leads to gaming.

The Dilbert comic strip may seem ridiculous, but as is always the case in Scott Adams’ cartoons, absurdity reflects reality to an uncomfortable degree (many of his cartoons are based on real-life examples submitted by readers). Incentives can have unintended consequences, often encouraging employees to behave unethically. For example, if you were earning a subsistence wage as a packer for Green Giant, and the company announced that a bonus would be paid to every employee who could find and remove insect parts from packages of frozen peas, what would you do? Possibly what many of the employees did—bring insect parts from home to earn the incentive.

There are, of course, more troubling examples of the dark side of measurement-based motivation. In the sad story of system-wide cheating in Atlanta Public Schools, 178 employees, including both teachers and principals, are now suspected of inflating scores on standardized tests to earn the significant rewards that come with rapid improvements in school performance. Outright swindles, such as Bernie Madoff, are all too common.

In sales departments there are more subtle examples of gaming incentive systems. Sales departments have been known to count all sales in the current quarter toward commissions—even though many of the sales are not actually closed. 

Conclusion

It’s dangerous to rely too much on measures for motivation: The more you emphasize measures, the more apt the measures are to cause dysfunctional, even unethical, behavior.  If you need to use measures for accountability and incentives, be careful.  Measures can’t replace management; they are a management tool.  It is necessary to make sure that the measures are reasonable – not gamed – and that accountability is understood and positive. 

Comparison, Context, and Connotation: Turning Data into Insight

How can we turn inert data into dynamic insights? We turn measurement into data, data into information, information into meaning, and meaning into insight. Each of these four steps is critical. If you are building talent measurement systems or using talent measurement as part of your management responsibilities you should know the three key Cs.

Turning Data into Information: The Key of Comparison

Many organizations have generous amounts of data filling their databases, but few organizations make use of it. Data by itself is meaningless. And at this point, it’s clear that our processing skills haven’t kept pace with our ability to collect terabytes of it. 

Imagine looking at a column of numbers. What does it tell you? Nothing, I imagine.

Now imagine that you can look at the same column of numbers with other data points for comparison. You might compare each piece of data to:

• Time (each data point is part of series across time)
• Employees (each data point is one employee’s performance)
• A norm (one number represents typical performance, and another number represents actual performance)
• A goal (one number is the target, and the other number is the actual amount)
• An implication (one number is an employee’s performance, the other number is an incentive payment associated with it).

Data is useless unless it is compared to something. Trends matter. 

The need for comparison isn’t always obvious, however. Occasionally, I’m asked to analyze an employee survey and there’s no good way to compare the data. Often, data is collected without considering how it will be turned into information. you have to compare across time, subgroups or at least across different questions. To turn data into information, you need a comparator.

To understand a survey, comparator can be previous survey results, a goal, normative responses, or the results of measurement in similar organizations (the basis of benchmarking).   In a pinch, you can compare survey questions to each other, but this provides limited information.

Turning Information into Meaning: The Key of Context

Information, however, is also never useful in itself. To have meaning, it needs to  be placed in context and interpreted. It’s critical to ask why.

• Why does the pattern of numbers vary with time?
• Why do employees’ levels of performance vary? Are there differences in aptitude, skill, or motivation? Or are they working in different environments?
• Why is our data above or below the norm?
• Why are (or why aren’t) we achieving the goal?
• Why am I receiving a smaller bonus than other employees?

In many situations, meaning is elusive because it requires a broad understanding of context. If two employees have very different performance results, are they working in the same context? Does one employee have more difficult tasks—a more involved project, a larger territory, more complex machinery to run? Only when you’ve determined that the context is comparable can you infer that different levels of skill or motivation underlie the difference in results.

Management is drowning in information. As we collect more and more data, comparison becomes easy. But, putting information into context requires bridging different data sources, integration, and creative thinking. The strategic and operational environment matters. Important information is relevant, given the context. 

Turning Meaning into Insight: The Key of Connotation

Given the process of translating data into meaning, it’s easy to see why there is miscommunication. Two people looking at the same data can have completely different interpretations—and they may not even know it. 

This may seem paradoxical at first; after all, measures are precise way of communicating. We wouldn’t say a car “doesn’t need much gas.” Instead, we use mathematical precision to talk about miles per gallon. Imagine the reactions of shareholders and analysts if an executive talked about “a pretty good investment,” rather than discussing a percent return on equity.

Nevertheless, people with different perspectives or goals see measurement results very differently. As with other types of communication, measures have denotative and connotative  meaning. Making a distinction between the two types of meaning is the key to understanding measures.

To illustrate the difference, let’s look at the term re-engineering. In literal (denotative) terms, re-engineering is a way to understand business processes and optimize them. In subjective (connotative) terms, there are implications of re-engineering—mass layoffs. I once used re-engineering as an example in a speech that I was giving in a company and received a strange and hostile response.  It turns out that the there was a history of using the term euphemistically!  That speech never recovered—the mere use of the term destroyed any trust between the audience and me.  

In measurement, the denotative meaning is often defined mathematically. It is the connotative meanings, however, that often matter to employees. In other words, the implications of the measures are more important than the measures themselves.  

It’s important to remember that the implications of measures are personal. How employees interpret measures, and how they react to performance appraisals, are influenced by their upbringing, their personalities, their motivations, and their worldviews. 

For example, if a salesperson is focused primarily on money as an indicator of success, he may consider performance measurement only in terms of the size of his bonus. It’s likely this will lead to misunderstandings with a company executive, who is looking at the measures to answer different questions: Does the salesperson need more training? Is the product “good.” Did the customers have good experiences?

In another common example, it may not be possible for an employee to receive measurement-based feedback constructively for any number of reasons. If the employee is perfectionistic or competitive, she may only be able to receive the feedback as criticism. Another employee may be so consumed with feeling miserable about failing to meet last year’s goal that he can’t engage in an authentic conversation about the future.

In my experience, many employees will try to avoid measurement because they distrust management, or are afraid of being targeted in a blame-oriented culture. This example of connotation is probably all too familiar to the readers of this blog.

Of course, personal perceptions and assumptions can be influenced. To build meaning, and ultimately insights, organizations must spend time decoding both the denotative and connotative meaning of measures. Unless both the denotations and connotations are addressed, there is little chance of communicating with the measures successfully, or gaining insight from the data.

Decoding measures is a dialogue. As a consultant, I need to discuss both the objective and subjective implications of measures with my clients. Managers need to do the same with their employees. It is only through dialogue that the measures’ contrasts and contexts, the meanings and implications, can be understood.

This is the path from data to shared insights. Of course, the path is littered with suggestions from many sources. But the salient point remains: To reach shared insights from measurement, organizations need to confirm that there is shared meaning. We can think about this as a four-step process:

There is no data without measurement, no information without data in comparison, no meaning without an understanding of information in context, and no insight without communicating shared meanings.

Is your organization taking steps to make sure that insights are gained from measurement?

In the next posts I’ll talk about motivation, organizational learning, and accountability. In the meantime, I welcome your thoughts.

Charley Morrow

Human Performance Measures: Start of a Series

I’ve been working with people measures for more than 25 years. Nearly every day, I see strong reactions to these common leadership tools.  Some embrace measurement as a tool for positive change, and others are nervous. Some question the measures, and others hide behind the authority of the data. 

These reactions to measurement and data fascinate me. They also hold the key to getting results from measurement systems. 

When measurement systems work well, people develop understanding, gain insight, become motivated, and set new directions. Just as often, however, measures simply do not work. In these cases, people ignore the measures or build elaborate defenses to dodge, manipulate, or diminish the data.

Over the next few months I’ll be writing about how systems and people respond to measures of human performance and how organizations can get beyond negative reactions. This is a topic I’ve been researching for years, and it may be my strongest and most nuanced area of understanding. 

I started my career focused on measurement systems. I took enough graduate courses in statistics and methodology to work as a psychometrician, and my dissertation combined the disciplines of psychology and economics. 

As I matured and worked in the real world of organizations, I started to see that the value of measurement can be found less in precision and mathematical finesse than in communication and learning. The most elegant performance management system is useless unless it is genuinely called on to help people communicate, learn, and adapt. 

In other words, measures need to be applied to produce data; data needs to be reviewed and interpreted to be useful; and useful information needs to be considered in context if people are to learn and improve. 

I can say with confidence that measures and data alone will not change organizations or behavior. There are too many psychological, organizational, and social factors that can prevent measures from translating into learning and improvement.

As a society, we spend huge sums of money on human performance measurement—and we start measurement early. All of us are familiar with the U.S. public education system, which now tests every student in the third through eight grade annually. In a number of states, databases are being developed to link these test scores to school, teachers, and student demographic information. 

When we graduate from the public education system, we find that most large organizations rely on annual employee appraisal systems. A manager can spend a few months each year rating employees, summarizing the information, and providing feedback. 

Despite the intensity of the data-gathering, improvement is not obvious. Many are dissatisfied with the measurement systems.  As a result, these measurement systems are often re-imagined and implemented with great hope and promise, only to fail. I don’t think much of this activity and investment. Don’t misunderstand: I’m a fan of measurement, because it’s critical to precise feedback and growth. But I’m an advocate for thoughtful investment in measurement. I’ve seen its transformative power. 

The public education system is still experimenting with measurement systems, and will be for years to come. Some corporations rethink their annual appraisal systems regularly. 

Technological and social trends suggest that performance measurement will only increase. Some argue that this investment is inappropriate. Addressing the merits of this societal investment isn’t my purpose here. My purpose is to make sure that individuals, organizations, and society get more value from the investments that are made.

I have workable tools and tips to make sure all of this data yields some return. Paradoxically, I won’t spend much time writing about measures. As I’ve said, it’s not as much about the measures as how they are used. I hope you will find the posts in the following weeks useful.