Myths of Measurement: Talent Measures Are Unaffected by Context

Physical measurement is barely affected by context: an inch is always an inch, and a cabinet always has the same dimensions, whether you are building it, improving it, or removing it.

This is not true of talent measures; talent measures are extremely sensitive to context. The same measure will yield different results in different contexts—whether you are selecting, developing, or laying off employees. It’s not a good idea to assume that you can use one talent measure for different purposes.

To understand the prevalent myth that talent measures are unaffected by context, we need to understand that measurement is just a method of conveying information: that is, measurement is a language. While the mathematical language of measurement is more precise than spoken language, meaning will vary with context.

Consider a competency rating. If the measure is used to set compensation many will only see the measure as a gateway to pay. If the organization uses the measure for two different purposes—compensation and developmental coaching—the coaching context will be contaminated by the context of pay. When it comes to competency ratings, employees often pay more attention to the context than to the measure itself.

Context is often more important than the measure. Let’s look at a few examples.

The Context of Performance Management

Most organizations have an annual performance appraisal. In most cases, an organization will review the number of ratings at each point on the scale (the distribution). Given obvious variability in performance, we would expect a normal distribution—a few employees would receive high ratings, and a few low ratings, but the great majority would cluster around the middle of the scale. In most organizations, however, nearly all the employees are clustered at the top of the scale, and only a few fall near the bottom. The distribution is skewed.

Over the years, skewed performance ratings have caused consternation, difficult conversations, and organizational chaos. Executives have looked at the distribution of performance ratings and thought:

• We sure have a great workforce—everybody is doing well!
• This measure is obviously biased—I know our workforce is not that great.

It should come as no surprise that the ratings are skewed, considering the context. Because the ratings may affect compensation, bosses tend to give higher ratings. The social context, not the measurement process, is causing the skew. Nevertheless, many organizations look for a better measure to provide more differentiation between employees, or to increase the number of low-rated employees. 

No matter how many times you change the performance appraisal measure, you’re unlikely to get a different distribution. The context stays the same, and as a result, the distribution is likely to stay the same. The employee/boss relationship will lead to a preponderance of positive ratings, and changing the measurement system will never solve the problem of skewed performance appraisals.

Solutions such as forced ranking, which I have discussed elsewhere as inappropriate, are simply masked attempts to develop a better measure. They may change the distribution, but they suffer from other problems such as spurious differences.

In any organization, the solution to skewed ratings in performance appraisal won’t be a better measure. Certainly, it is easy to change the measure.  Further, there are many different tweaks that can be made, including different rating scales, number of points on the rating scale, different dimensions to rate.  If, however, you really want to change the distribution, develop better management discipline and use the existing measurement system. This requires discipline difficult conversations between bosses and the employees who work for them. 

The Context of Employee Engagement

There is currently a small revolution happening in employers’ views of employee engagement. Starting with Marcus Buckingham’s research linking engagement survey results to positive outcomes such as productivity, customer satisfaction, and employee retention, employers have rediscovered employee surveys. Many executives worry about an unengaged workforce and the impact on their business, and many employers are surveying their workforce for the first time.

Some organizations have even linked incentives to engagement measures—for example, by increasing or decreasing a manager’s compensation based on the engagement scores in his or her area. This practice seems justified, since we can find relationships between engagement and outcomes such as profitability and retention. Anything that can be done to increase engagement should be tried.

As with performance appraisals, however, adding financial incentives will fundamentally change the context of the measure. Employees have told me, in confidence, that their manager asked them to respond to the survey positively, regardless of how they were feeling. One of the most disengaging things a manager can do is to ask an employee to misrepresent herself. The effect will be an extreme form of contamination of the measure. While actual engagement will decrease, the measure will show an increase. This is a form of cheating.

Another Name for the Myth

Psychologists who work with performance measures have developed a term for how measures are changed by context: When a measure of performance is affected by non-performance factors, they refer to it as criterion contamination. Because the performance variable will be contaminated by the context, researchers are warned not to use performance appraisal results when conducting research. If they do, the research will not yield meaningful results.

The various social and motivational forces that affect performance appraisals are one example of context. There are many other examples of contextual influence: organizational culture, business processes, personal beliefs, discipline, and so forth. These contexts affect every type of talent measures.

Organizations often forget about criterion contamination and try to use a single measure for different purposes. If a measure has been linked to incentive pay, for example, it’s not possible to use the same measure to study the relationship between employee engagement and customer satisfaction. The measure has been contaminated by the compensation context, and the context is always more powerful than the measure.

In this case, the solution to criterion contamination is to get a new measure. It’s certainly inconvenient to develop additional measures, especially when a perfectly good measure already exists.

The Same Context May Be More Different Than You Think

As the engagement example above shows, just as the meaning of measures varies according to the organizational context, it also varies according to individual context. This is another aspect of the myth of unaffected measures: there is often an assumption that the measure means the same thing to you and me.

As discussed in previous posts, this is the connotative, or subjective, meaning of the measure. Although in a denotative sense the measure will have exactly the same meaning at any level of an organization, within that organization, the measures will mean radically different things to different groups and different individuals.

In the engagement-gaming example above, for example, the context of the measure varies between the different parties:

• Executive management is concerned with engagement and its impact on the business in terms of productivity, customer satisfaction, or employee engagement 
• Supervisors have incentive compensation and are concerned with how the scores will affect their pay 
• Employees feel pressure to respond positively, but may have insights to share—once again the system is preventing them from having a voice.

Recognizing the existence, and the effect, of connotative meanings presents one of the biggest challenges in talent measurement. If we pay attention to the connotative meanings—that is, the individual and group contexts surrounding a measure—we can communicate to create shared meaning. In a culture of open communication, there is a significant opportunity to get more value from measures.

To Use Measures Well, Remember the Myth

Organizations need employees who are engaged in achieving organizational goals. This idea goes by many names, such as ownership culture and results orientation. Measures are often used to encourage engagement, with the intent of building a shared worldview and an understanding of the organization.  Performance appraisals and scorecards help keep everyone on the same page. Or do they?

It’s important to remember how easily measures are changed—some would say corrupted—by context. Misuse a measure once and employees will remember it for a long time. Use a performance appraisal for laying off employees, and this will change the context in the future. It’s easy for a measure to pick up new connotations.

Human resource departments and leaders have an opportunity to manage the meaning of talent measures at all levels of an organization. One way to do this is to watch for this myth in action. Remembering that every talent measure is affected by context can lead to a more discerning use of measurement, better communication, and, ultimately, more positive outcomes.

Of course, I’m not the first to point out this challenge. In 1975, Donald T. Campbell observed a methodological phenomenon that some refer to as Campbell’s Law: 

The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

In the nearly 40 years since Campbell’s observation, talent measures and performance metrics have proliferated in organizations. The proliferation is accelerating as measurement becomes inexpensive and accessible. In my experience, however, few consider this dark side of measurement.

In the next blog post, I’ll consider how talent, which Campbell refers to as “the underlying social process,” is affected by measurement.

Mistrust of Talent Measurement

In many organizations, you’ll find fear and mistrust. You’ll also find that employees often focus these negative sentiments on the organization’s measurement system.

A measurement system won’t work without trust. Trust is the basis of functioning human communication, and as I’ve discussed in previous posts, measurement is best understood as a means of communication. In this blog post we’ll explore these issues, and look at how to increase trust in measurement.

I had a client with a world-class performance management system. The system included many best practices:

• Individual goals cascading from the strategy of the organization
• Employees and managers met at the beginning of the year to discuss goals
• Coaching happened throughout the year, with a formal documentation period
• Final evaluation happened at the end of the year, with ratings of behavioral competencies as well as objective performance metrics.

The  employees hated the evaluation system. Why? They didn’t trust it. This organizations performance management system, like many, was a de-motivator!  

When there is no trust in measurement, the results are often ugly. In pre-revolution feudal France, the physical measurement system was controlled by the aristocracy. Measurement was idiosyncratic, fragmented, and non-standardized; there were approximately 14,000 different units of physical measurement. Mistrust of measurement, or measurement-based decisions (such as those involved in commerce), was one of the factors exacerbating class tensions.

There are historians who report metric riots over measurement issues during this period. When the nobility surrendered their privileges after the storming of the Bastille, these privileges included giving up control of measures.

This may sound extreme, but is it? What if you couldn’t trust that a pound was a pound, and the person measuring your pound of pasta was defining the measurement?

Today, we assume that physical measurements are accurate and standardized—everything from a pound of pasta to land surveys. There are governmental bodies regulating physical measures. We rarely even think about the physical measurements that we use daily.

The same is not true of talent measures. Each organization has its own set of talent measures (for good reason), and these measures are typically controlled by management.

As in the French revolution, those who control the measures have a significant source of power to use and, in some cases, abuse. Before the revolution, serfs often felt they were being cheated by nobility.  They probably were. After the revolution, there was a movement to base measurement on universal, naturally occurring objects. This is the origin of the metric system.

How does this relate to my client with the world-class performance management system? Employees hated the system because it was unpredictable and apparently arbitrary. It was unintentionally failing the fairness test in two ways. 

First, employees saw the performance appraisal procedure as unfair because it was constantly changing. In an attempt to improve the system, senior management was constantly tweaking it. Supervisors and employees weren’t sure how the system would ultimately appear on the intranet. Little was done to communicate how the system worked, or why it was changed. Given the complexities of the measures and the misunderstanding of the process, it’s understandable that employees started to wonder whether something nefarious was going on. Sometimes the tweaks made winners and losers. As a result, a near revolution was brewing in what Elton Mayo would have referred to as a social system. 

Second, employees saw the system as unfair because of the way rewards were distributed. The pool of bonus money was spread among the employees according to the ratings, often as the final ratings were calculated. This is not an unusual profit-sharing plan. Unfortunately, there were last-minute changes, sometimes for departments as a whole, that had unexpected effects on ratings and compensation.

Stability and transparency is needed to make a performance management system trusted and functional. This requires an acceptance of the imperfections of measurement, and much more communication. To a large degree, tweaking will not improve a talent measurement system—it will only serve to further distort the meaning of the measures. Often, tweaks occur because we believe in the myths of measurement:

• My mental model of performance is correct
• Measures are real
• There is a perfect measure
• People and the measurement system do not change in the process of measurement.

Opposition to Measurement

This brings up a second puzzle for a measurement guy like me. Why are so many employees, and unions, opposed to measurement?

Coming out of graduate school, I understood the formal side of measurement very well. I was surprised at the level of animosity toward measurement. I naively assumed that the honesty and accuracy associated with good measurement would enhance relationships, including employee and labor relations. I soon learned that in practice there is little appetite for measures, mostly because of how they have been, and are, used in organizations. 

I’ve found four major reasons for this opposition:

• Employees being held accountable for factors beyond their control
• Arbitrary use of power
• Differentiation between employees
• History of Taylorism.

These four reasons hamper effective talent measurement. They exert such a powerful force that new measurement initiatives start with a lack of trust. Often this lack of trust must be addressed and overcome before organizational learning and employee motivation can happen.

If you know of additional reasons, I’d love to learn about them.

Inappropriate Accountability

As I discussed in a previous post, measures are contaminated by factors beyond an individual employee’s control. For example, low-performing teammates may prevent an employee from reaching her full potential. This is, of course, a fact of life—we can’t control everything.  When employees are faced with a measurement system that ignores factors beyond their control, they can feel that their efforts are futile, and the measurement arbitrary. This tends to erode trust in the organization and management.

Arbitrary Use of Power

Managers have the power to change the measurement system; sometimes this power is used to withhold rewards or punish employees in some way. For example, if a salesperson has earned a large reward based on a previously defined incentive system, sometimes the system is reconfigured to avoid the large payout. Of course, this breaks trust—and it does happen. Unfair actions related to measurement  are remembered for a long time.

In addition, management is often responsible for interpreting or framing measures. Setting unattainable goals, for example, can lead to employee dissatisfaction and decreased morale, as employees give up trying to improve their performance.

Differentiation between Employees

Measurement is often used to differentiate between employees. If you don’t trust the measurement system to make important distinctions related to employee performance, you won’t like differentiation. 

In the coming year, many U.S. school districts will differentiate between teachers using standardized tests. Some will think this is fair, but many will not. Methods have been developed, such as value-added scores, that attempt to statistically control for factors, such as race or wealth, that affect student growth. Despite this, many fundamentally mistrust the standardized tests on which the evaluation systems rest.

Differentiating between employees is also antithetical to unionist philosophy. Unions believe that differentiation reduces solidarity, and they tend to believe that everyone is the same. In fact, this mistrust of measurement, based on valuing both similarity and solidarity, leads to a question often asked by unionists: How can we tell who is better than another? If you can’t answer this question, the only basis for differentiating payment is seniority. While experience does increase knowledge, skill, and even wisdom, it’s not performance. It is, however, an unambiguous differentiator.

History of Taylorism

In a sense, Frederick Taylor’s use of time-and-motion studies to set performance goals destroyed trust in measurement. As I discussed in the last post, Taylor’s approach to measurement was used mechanistically, against employees, and not in partnership with them.

Interestingly, Taylor assumed that employees would embrace scientific management. Sometimes they did, but most often they didn’t. He was, of course, assuming that employees were motivated only by money and the possibility of higher wages based on increased work.  Of course, we now know that this assumption is wrong.

In an interesting turn of history, employee unions and strikes were ultimately the downfall of Taylorism. Strikes by public sector employees caused Congress to hold hearings. After five years, scientific management was essentially outlawed by limiting the use of incentive wages and stopwatches.

Building Trust in Talent Measures

So how do we build trust in talent measures, given that the way measurement has been used in the past creates negative preconceptions? I have three suggestions that will help create a climate of fairness and transparency:

• Make measurement predictable. Build the best technical system you can, and accept that some people will manipulate the system. The solution to manipulation is rarely a better measure—most often, it’s better management and leadership, using the measures.
• Communicate, communicate, communicate! Communication is creating shared meaning. An organization’s definition of the meaning of measures will not be the same as employees’ meanings, or experience. Both meanings need to be acknowledged and managed in a two-way give and take. Never assume that a measure means the same thing to everyone.
• Build trust in measurement. Anything that could be interpreted as using the measures against the workforce or individuals should be avoided.  Apply measurements consistently across your organization. Hold everyone accountable to the same standards, and make those standards clear.  Remember, we want our employees to be thinking about the work and engaging with the organization.  We do not want them to be thinking, “Is this measurement system fair?”

Talent Measurement Schools of Thought

Here is a puzzle: In our day-to-day life we do not treat people as inanimate objects—but we try to measure them as if they are! We treat the people in front of us as living, breathing, reacting entities, but few consider the complexity and reactivity of human nature when developing or managing with measures. Why the inconsistency?

Two Schools of Thought: The Taylor and Mayo Dichotomy

To solve this puzzle, you have to go back to school—graduate school. As a graduate student, you’ll probably learn one of two different approaches to talent measurement. One school of thought is focused on the technical aspect of measurement, and the other on the human aspect. The challenge for measurement professionals is to master both schools of thought. The two are rarely reconciled, however. Professionals generally have expertise primarily in one approach.

The technical or engineering school will teach you how to calculate reliability and validity, and introduce you to different measurement methods. This school of thought dates back to Frederick Taylor, one of the first manufacturing engineers. Frederick Taylor is considered the father of scientific management, which emphasizes task analysis, efficiency studies, time-and-motion studies, and using compensation schemes for motivation. 

The human relations school has a different point of view: Employees are complicated, and don’t work mechanistically. If your graduate program emphasizes human relations, you’re likely learn more about personality types or team functioning measures that will facilitate interactions between people at work. You’ll be introduced to validity and reliability, but you’ll be taught very little about the technology and theory of measures. The human relations school of thought dates back to Elton Mayo, a psychologist. 

The ghosts of Taylor and Mayo haunt today’s organizations. To this day, consultants, managers, and leaders adhere to one school or the other. Taylor adherents tend to advocate for measurement as a formal and rigid process. Mayo adherents focus more on group processes, interpersonal communication, and intrinsic motivation. 

Both Taylor and Mayo made essential contributions to the art of management and leadership. But it’s not an either/or choice. It often takes decades of experience to merge the two schools of thought into a practical working knowledge of measurement. Some never see the dichotomy and its implications.

I’m writing this blog post in the hope that we can accelerate the process of combining and ultimately uniting these two schools of measurement.

The Engineer: Frederick Winslow Taylor (1856 – 1915) “In the past the man has been first; in the future the system must be first.”

Taylor grew up affluent and gifted in the second half of the 19^th century, in an era of huge industrial change. He chose not to follow his father into the legal profession, although he was accepted into Harvard. Instead, he worked in industry, starting as a machinist and becoming a foreman, and went on to study engineering. 

As an engineer, he first improved manufacturing technology such as lathes and forging equipment. Early on, he noticed that these technical improvements demanded similar organizational innovations to be effective. As his ideas developed, he saw manufacturing as a larger system that could be improved by optimizing the various pieces to contribute to the larger system. Over the course of his career, he contributed his ideas to equipment (he had several important patents), business processes (such as accounting methods), and methods of managing employees.  

Taylor and Time-and-Motion Studies

As he looked at the larger manufacturing picture, Taylor was concerned that laborers were not working at full capacity. To fix this problem, he identified the optimum work-output level, and provided incentive pay for this level of output.  

Determining workers’ optimum output involved time-and-motion studies. Taylor divided the work into steps, each of which he timed separately. He then combined the time for each step into a total time for the job. By dividing the work day by the total job time, he arrived at an optimum production rate.

Workers were paid on a graduated scale. Low levels of output were paid very little, but as productivity approached the maximum, unit pay increased. Workers attaining the optimum production rate would be paid 60% more using Taylor’s methods. 

While he became infamous for his time-and-motion studies, it’s important to recognize that, for Taylor, these studies were part of a larger system of managing employees. Taylor used worker productivity as a talent measure. He studied measures of productivity to make decisions, organize work, set production expectations, motivate employees, and identify employees to retain. In the best cases, Taylor’s scientific management methods could reduce costs and increase productivity by 50% to 100%.

Human reaction to measures and management methods didn’t factor into Taylor’s thinking. He was convinced that employees only work for money. Labor problems were simply an engineering challenge to be managed. Taylor paid lip service to selecting and developing talent—he mostly set output targets. Workers who were able to keep up the pace self-selected and developed their capability.

Taylor’s blind spot—the human factor—can be seen in many contemporary organizational improvement interventions, such as re-engineering, which has a success rate as low as 30%. Human readiness and acceptance of change is often a barrier to re-engineering success.   

Taylor’s approach also was inconsistent. Sometimes it worked, sometimes it led to significant problems.

Employee reactions to Taylor’s intervention often led to work actions and strikes.  Ultimately there was an congressional investigation. By the time of Taylor’s death at age 59, Congress had outlawed use of stopwatches and bonus payments in the federal government. Scientific management was increasingly discredited.

The Humanist: George Elton Mayo (1880 – 1949) “So long as … business methods take no account of human nature … expect strikes and sabotage to be the ordinary.”

Mayo grew up in a distinguished Australian family. He began his studies in medicine and ended up studying psychology, focusing on social interactions at work. His most famous research work can be found in the Hawthorne studies, which demonstrated that employees are largely influenced by social factors, and that they react to being observed.

Mayo’s most important work coincided with the Great Depression. He believed that the industrial revolution had shattered strong social relationships in the workplace, and he found that workers acted according to sentiments and emotion. He felt that if managers treated workers with respect and tried to meet their needs, then both workers and management would benefit.  

Mayo’s research indicated that belonging to a group is a more powerful motivator than money. In his management philosophy, he saw attitudes, proper supervision, and informal social relationships as the key to productivity.

Some consider Mayo’s work to be a reaction to Taylorism. But Mayo was also concerned with output and productivity. Unlike Taylor, however, he was interested in the social and psychological interventions that increased productivity. These interventions are indeed helpful, and understanding the human factor is critical.  

Thanks to Mayo’s work, we recognize that, in organizations, informal social structures matter as much as formal structures, such as the chain of command. For example, a likeable senior engineer who dislikes a new manager could undermine the manager’s authority by making jokes at his expense during every meeting. In effect, the engineer becomes more influential than the manager—outside the hierarchy of the organizational chart.

Today, many organizational interventions emphasize team-building, and are based on the recognition that organizational culture is important, and managers have ongoing relationships with employees. By acknowledging the importance of the informal structure of an organization, factors such as relationships, informal leadership, and influence can be aligned with organizational needs and direction.

Mayo’s insights were synthesized into a school of thought referred to as human relations. The human relations school continues strong to this day, often in the form of leadership development, team building, or change initiatives. 

The insight missed by Mayo is that measurement—even Taylor’s productivity measurements—are essentially a social process. Measurement is simply a method of communication—a way to make meaning between groups.

Since Mayo, many people have failed to make this essential connection: We can extend Mayo’s insight into the importance of informal (social) structures into an understanding of the importance of the informal (connotative or personal) meanings of measures. As I have discussed before, the informal meanings of measures matter as much as, if not more than, their formal meanings. Like social structures, these connotative meanings can be managed—but only when their existence and importance are acknowledged.

If you’re creating an organizational, and you follow Taylor, you may believe that compensation is the sole motivation for performance and advancement. If you follow Mayo, you may believe that love, fear, and other ineffable human factors are the primary motivators. 

In the same way, the designers of a formal measurement system may believe that their measures will motivate by providing people with a positive opportunity to make more money (the denotative meaning). Instead, the designers may find that the connotative meanings provoke reactions that ultimately trump their intentions—reactions from outright rejection to gaming the system.

It’s odd that many of our measurement systems haven’t progressed beyond Taylor’s way of thinking. We have many tools to address the informal meanings of measures—tools we can draw on from interpersonal communications theory, management practices, and organizational learning.

Another danger of following Mayo’s approach is that it often pays too much attention to the informal and emergent social structures of an organization. While these informal structures are powerful influences on individual performance, it is possible to merge formal and informal organizational structures into a shared structure. This is where measurement can be incredibly effective, if it is used as a means of communication: It can create shared meaning that bridges the organizational and personal definitions of performance, motivation, and reward.

Finding a Middle Ground

What’s most unfortunate about the Mayo vs. Taylor bifurcation is that they were both right: The difference between the two schools of thought is ideological, not practical. In practice, we use both approaches. We need both engineers and social scientists (psychologist and sociologists) to run organizations efficiently.

If we attend one graduate school, we may learn to develop measures that are technically good, but we’ll have trouble assessing the human reaction to measurement. If we attend another school, we may learn to facilitate social interaction and meaning, but we won’t be trained to motivate, direct, or improve performance through measurement and feedback. Personally, I attended a more technical school, but my life and work experiences have led me to appreciate a balanced approach.

What Taylor missed was the importance of social structures in motivation, and the human factor in reaction to measurement. What Mayo missed was that measurement in itself is a social process, and measures have informal (social) meanings that can be managed.

Today, 80 years after Mayo’s Hawthorne studies, we should be able to merge the two schools. There is a wealth of possibilities for applying Taylor’s ideas in measuring individual productivity. At the same time, we’ve vastly increased our understanding of human relations—there’s a huge industry that’s evolved out of Mayo’s original insights.

Finally, in resolving the polarity of these two approaches, we need to acknowledge that measurement is communication, and that communication is shared meaning. By starting with a simple point—that people always react to measurement, and that the reaction is unpredictable—we can take the denotative and connotative meanings of measures, the formal and informal structures in organizations, and the two schools of thought, and synthesize them into an elegant, effective approach to talent measurement.