When a measurement becomes a target, it is no longer a valid measurment

The title of this essay is one version of an insightful observation made by various people in a variety of settings .

The first time I became aware of that general thought was when I read about the Westinghouse Effect which is that the observation of an event is influenced by the act of observation.

Two eponymous designations refer to formulations that expand that concept and give it more useful meaning; Goodhart's Law and the Lucas critique. These have important implications in this era of hyperplastic medical guidelines, pay for performance, quality measures which pretend to measure the unmeasurable and the cookbookization of primary care medicine.

Charles Goodhart was an economist in Great Britain who expressed the following thought:

Once a measure is made a target for the purpose of conducting policy, it will loose the information content that would qualify it to play such a role.

A medical example is the four-hour pneumonia rule. There are data indicating that those pneumonia patients who received antibiotics within some several hour time frame did better that those whose antibiotics were delayed. This even conforms with common medical sense-a patient with a serious infection requiring antibiotics should do better getting the medication sooner rather than later. So promptness of delivery antibiotics was considered to be a measure of quality and then became a target. Once a target, ER personnel seemed to have treated towards this target and later we learned that one unintended consequence was some not insignificant number of patients were given antibiotics within the magic time frame but did not have pneumonia at all.

This line of thought was developed further by another economist and Nobel prize laureate, Robert Lucas.

Lucas said it was naive to think one could predict the effect of a policy purely based on aggregated historical data. An example might be the data suggesting promptness of treatment influenced pneumonia outcomes.To predict the outcome of a policy change one has to consider how individuals are likely to behave given the change.

To predict the effect of a policy change (rewarding 4 hr treatment for example) you need to consider the constraints the players operate under and basic human nature.Outcomes may change ( even if those outcomes are not measured) when policy ( or rules of the game) is changed.The effort to meet some deadline that folks are graded on may well take away efforts to provide necessary timely care for other patients not currently covered by some quality rule and target.In the era before the imposition of the four hour rule, promptness of antibiotic administration may have been one of a number of indicators of a good general care. Once it became a target it lost its value as an indicator of quality even though now it was considered to be an official indicator of quality.

This general theme was discussed recently by the Management editor of The Guardian.See here.

He offers this example also from the medical world having introduced a concept analogous to Gresham's Law, bad measurements drive out good ones.

What happens when bad measures drive out good is strikingly described in an article in the current Economic Journal. Investigating the effects of competition in the NHS, Carol Propper and her colleagues made an extraordinary discovery. Under competition, hospitals improved their patient waiting times. At the same time, the death-rate following emergency heart-attack admissions substantially increased. Why? As targets, waiting times were and are measured (and what gets measured gets managed, right?). Emergency heart-attack deaths were not tracked and therefore not managed. Even though no one would argue that the trade-off - shorter waiting times but more deaths - was anything but a travesty of NHS purpose, that's what the choice of measure produced.

At the risk of underlining something too much-the point is that measurements loose their value as measurements once they become targets and that does not depend on the original validity of the measure. It can be a great measure, but it is ruined when it become a target. So the answer is not to get better measures much as the answer to bad government is not to replace the officials with better ones.

Goodhart and others made this observation some time ago but fortunately we in medicine were- for quite a while- spared the pain of seeing it first hand in our practices.