Myths of Measurement: Talent Measures Are Unaffected by Context

Physical measurement is barely affected by context: an inch is always an inch, and a cabinet always has the same dimensions, whether you are building it, improving it, or removing it.

This is not true of talent measures; talent measures are extremely sensitive to context. The same measure will yield different results in different contexts—whether you are selecting, developing, or laying off employees. It’s not a good idea to assume that you can use one talent measure for different purposes.

To understand the prevalent myth that talent measures are unaffected by context, we need to understand that measurement is just a method of conveying information: that is, measurement is a language. While the mathematical language of measurement is more precise than spoken language, meaning will vary with context.

Consider a competency rating. If the measure is used to set compensation many will only see the measure as a gateway to pay. If the organization uses the measure for two different purposes—compensation and developmental coaching—the coaching context will be contaminated by the context of pay. When it comes to competency ratings, employees often pay more attention to the context than to the measure itself.

Context is often more important than the measure. Let’s look at a few examples.

The Context of Performance Management

Most organizations have an annual performance appraisal. In most cases, an organization will review the number of ratings at each point on the scale (the distribution). Given obvious variability in performance, we would expect a normal distribution—a few employees would receive high ratings, and a few low ratings, but the great majority would cluster around the middle of the scale. In most organizations, however, nearly all the employees are clustered at the top of the scale, and only a few fall near the bottom. The distribution is skewed.

Over the years, skewed performance ratings have caused consternation, difficult conversations, and organizational chaos. Executives have looked at the distribution of performance ratings and thought:

• We sure have a great workforce—everybody is doing well!
• This measure is obviously biased—I know our workforce is not that great.

It should come as no surprise that the ratings are skewed, considering the context. Because the ratings may affect compensation, bosses tend to give higher ratings. The social context, not the measurement process, is causing the skew. Nevertheless, many organizations look for a better measure to provide more differentiation between employees, or to increase the number of low-rated employees. 

No matter how many times you change the performance appraisal measure, you’re unlikely to get a different distribution. The context stays the same, and as a result, the distribution is likely to stay the same. The employee/boss relationship will lead to a preponderance of positive ratings, and changing the measurement system will never solve the problem of skewed performance appraisals.

Solutions such as forced ranking, which I have discussed elsewhere as inappropriate, are simply masked attempts to develop a better measure. They may change the distribution, but they suffer from other problems such as spurious differences.

In any organization, the solution to skewed ratings in performance appraisal won’t be a better measure. Certainly, it is easy to change the measure.  Further, there are many different tweaks that can be made, including different rating scales, number of points on the rating scale, different dimensions to rate.  If, however, you really want to change the distribution, develop better management discipline and use the existing measurement system. This requires discipline difficult conversations between bosses and the employees who work for them. 

The Context of Employee Engagement

There is currently a small revolution happening in employers’ views of employee engagement. Starting with Marcus Buckingham’s research linking engagement survey results to positive outcomes such as productivity, customer satisfaction, and employee retention, employers have rediscovered employee surveys. Many executives worry about an unengaged workforce and the impact on their business, and many employers are surveying their workforce for the first time.

Some organizations have even linked incentives to engagement measures—for example, by increasing or decreasing a manager’s compensation based on the engagement scores in his or her area. This practice seems justified, since we can find relationships between engagement and outcomes such as profitability and retention. Anything that can be done to increase engagement should be tried.

As with performance appraisals, however, adding financial incentives will fundamentally change the context of the measure. Employees have told me, in confidence, that their manager asked them to respond to the survey positively, regardless of how they were feeling. One of the most disengaging things a manager can do is to ask an employee to misrepresent herself. The effect will be an extreme form of contamination of the measure. While actual engagement will decrease, the measure will show an increase. This is a form of cheating.

Another Name for the Myth

Psychologists who work with performance measures have developed a term for how measures are changed by context: When a measure of performance is affected by non-performance factors, they refer to it as criterion contamination. Because the performance variable will be contaminated by the context, researchers are warned not to use performance appraisal results when conducting research. If they do, the research will not yield meaningful results.

The various social and motivational forces that affect performance appraisals are one example of context. There are many other examples of contextual influence: organizational culture, business processes, personal beliefs, discipline, and so forth. These contexts affect every type of talent measures.

Organizations often forget about criterion contamination and try to use a single measure for different purposes. If a measure has been linked to incentive pay, for example, it’s not possible to use the same measure to study the relationship between employee engagement and customer satisfaction. The measure has been contaminated by the compensation context, and the context is always more powerful than the measure.

In this case, the solution to criterion contamination is to get a new measure. It’s certainly inconvenient to develop additional measures, especially when a perfectly good measure already exists.

The Same Context May Be More Different Than You Think

As the engagement example above shows, just as the meaning of measures varies according to the organizational context, it also varies according to individual context. This is another aspect of the myth of unaffected measures: there is often an assumption that the measure means the same thing to you and me.

As discussed in previous posts, this is the connotative, or subjective, meaning of the measure. Although in a denotative sense the measure will have exactly the same meaning at any level of an organization, within that organization, the measures will mean radically different things to different groups and different individuals.

In the engagement-gaming example above, for example, the context of the measure varies between the different parties:

• Executive management is concerned with engagement and its impact on the business in terms of productivity, customer satisfaction, or employee engagement 
• Supervisors have incentive compensation and are concerned with how the scores will affect their pay 
• Employees feel pressure to respond positively, but may have insights to share—once again the system is preventing them from having a voice.

Recognizing the existence, and the effect, of connotative meanings presents one of the biggest challenges in talent measurement. If we pay attention to the connotative meanings—that is, the individual and group contexts surrounding a measure—we can communicate to create shared meaning. In a culture of open communication, there is a significant opportunity to get more value from measures.

To Use Measures Well, Remember the Myth

Organizations need employees who are engaged in achieving organizational goals. This idea goes by many names, such as ownership culture and results orientation. Measures are often used to encourage engagement, with the intent of building a shared worldview and an understanding of the organization.  Performance appraisals and scorecards help keep everyone on the same page. Or do they?

It’s important to remember how easily measures are changed—some would say corrupted—by context. Misuse a measure once and employees will remember it for a long time. Use a performance appraisal for laying off employees, and this will change the context in the future. It’s easy for a measure to pick up new connotations.

Human resource departments and leaders have an opportunity to manage the meaning of talent measures at all levels of an organization. One way to do this is to watch for this myth in action. Remembering that every talent measure is affected by context can lead to a more discerning use of measurement, better communication, and, ultimately, more positive outcomes.

Of course, I’m not the first to point out this challenge. In 1975, Donald T. Campbell observed a methodological phenomenon that some refer to as Campbell’s Law: 

The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

In the nearly 40 years since Campbell’s observation, talent measures and performance metrics have proliferated in organizations. The proliferation is accelerating as measurement becomes inexpensive and accessible. In my experience, however, few consider this dark side of measurement.

In the next blog post, I’ll consider how talent, which Campbell refers to as “the underlying social process,” is affected by measurement.