Monday, April 14, 2008

When a measurement becomes a target, it is no longer a valid measurment

The title of this essay is one version of an insightful observation made by various people in a variety of settings .

The first time I became aware of that general thought was when I read about the Westinghouse Effect which is that the observation of an event is influenced by the act of observation.

Two eponymous designations refer to formulations that expand that concept and give it more useful meaning; Goodhart's Law and the Lucas critique. These have important implications in this era of hyperplastic medical guidelines, pay for performance, quality measures which pretend to measure the unmeasurable and the cookbookization of primary care medicine.

Charles Goodhart was an economist in Great Britain who expressed the following thought:

Once a measure is made a target for the purpose of conducting policy, it will loose the information content that would qualify it to play such a role.

A medical example is the four-hour pneumonia rule. There are data indicating that those pneumonia patients who received antibiotics within some several hour time frame did better that those whose antibiotics were delayed. This even conforms with common medical sense-a patient with a serious infection requiring antibiotics should do better getting the medication sooner rather than later. So promptness of delivery antibiotics was considered to be a measure of quality and then became a target. Once a target, ER personnel seemed to have treated towards this target and later we learned that one unintended consequence was some not insignificant number of patients were given antibiotics within the magic time frame but did not have pneumonia at all.

This line of thought was developed further by another economist and Nobel prize laureate, Robert Lucas.

Lucas said it was naive to think one could predict the effect of a policy purely based on aggregated historical data. An example might be the data suggesting promptness of treatment influenced pneumonia outcomes.To predict the outcome of a policy change one has to consider how individuals are likely to behave given the change.

To predict the effect of a policy change (rewarding 4 hr treatment for example) you need to consider the constraints the players operate under and basic human nature.Outcomes may change ( even if those outcomes are not measured) when policy ( or rules of the game) is changed.The effort to meet some deadline that folks are graded on may well take away efforts to provide necessary timely care for other patients not currently covered by some quality rule and target.In the era before the imposition of the four hour rule, promptness of antibiotic administration may have been one of a number of indicators of a good general care. Once it became a target it lost its value as an indicator of quality even though now it was considered to be an official indicator of quality.

This general theme was discussed recently by the Management editor of The Guardian.See here.

He offers this example also from the medical world having introduced a concept analogous to Gresham's Law, bad measurements drive out good ones.

What happens when bad measures drive out good is strikingly described in an article in the current Economic Journal. Investigating the effects of competition in the NHS, Carol Propper and her colleagues made an extraordinary discovery. Under competition, hospitals improved their patient waiting times. At the same time, the death-rate following emergency heart-attack admissions substantially increased. Why? As targets, waiting times were and are measured (and what gets measured gets managed, right?). Emergency heart-attack deaths were not tracked and therefore not managed. Even though no one would argue that the trade-off - shorter waiting times but more deaths - was anything but a travesty of NHS purpose, that's what the choice of measure produced.

At the risk of underlining something too much-the point is that measurements loose their value as measurements once they become targets and that does not depend on the original validity of the measure. It can be a great measure, but it is ruined when it become a target. So the answer is not to get better measures much as the answer to bad government is not to replace the officials with better ones.

Goodhart and others made this observation some time ago but fortunately we in medicine were- for quite a while- spared the pain of seeing it first hand in our practices.

2 comments:

Clark Venable said...

A corollary to this is that what is true for the population is not necessarily true for the individual. Giving antibiotics within four hours may benefit the population, but it is another thing entirely whether this patient, with this presentation, and this pathogen, acquired in this way, would benefit even if antibiotics were admistered in two hour or six hours.

DrRich said...

Dr. Gaulte,

Excellent post. This is one reason that P4P shows such wonderful potential for covert healthcare rationing. When the feds or the insurance companies specify, through their P4P guidelines, what a physician must do during their 7.5 minute patient encounter, they are also specifying (by omission) those things the doctor must NOT be doing during that 7.5 minutes.

DrRich