The randomized trial-more on its limitations-this time from quality movement advocates

Several authors who have spoken out against the move to make certain forms of "alternative" medicine acceptable have recognized that the notion that the randomized trial trumps all other forms of evidence simply does not work in regard to therapeutic systems that are scientifically implausible. Recent articles can be found here and here, When something like homeopathy or Reiki flys in the face of basic scientific principles ,a RCT is not only unnecessary but actually has the potential for mischief as ultimately a false positive will find its way into the literature and being used to justify pure smoke.

In the March 12,2008 issue of JAMA there is a commentary asserting that the randomized trial also may be not the verification mechanism of choice to analyze and verify or falsify complex system changes such as those that are part of the quality movement.

The author, Dr. D M Berwick, CEO and President of the Institute of Healthcare Improvement, has written on this topic before. In this earlier article he relates a brief history of the framework
that has come to be known as evidence based medicine (EBM,) gives appropriate praise for its stellar accomplishments and then asserts that in some regards we may have gone too far.

We have transformed the commitment to "evidence-based medicine" of a particular sort into an intellectual hegemony that can cost us dearly if we do not take stock and modify it. And because peer reviewed publication is the sine qua non of scientific discovery, it is arguably true that hegemony is exercised by the filter imposed by the publication process.

He is saying that the "normative framework for judging the value of evidence " has become so dominant that any evidence adduced outside of the framework may be dismissed out of hand and never see the light of day in publication. For his perspective if a quality project is not validated by a RCT counts for nothing and this he believes is wrong.

Quoting now from the Berwick 's JAMA article;

Many assessment techniques developed in engineering and used in quality improvement-statistical analysis, simulations, and factorial experiments-have more power to inform about mechanisms and contexts than do RCTS, as do ethnography, anthropology, and other qualitative methods.

Some of the authors who have lead the crusade against non-scientific medical treatments (known now as woo) have recognized that the RCT need not be the trump card some believe it to be in the context of therapies that are clearly not scientifically possible. Here the RCT can really only be a false positive. At least some of the quality movements movers and shakers are arguing that in the context of complex quality improvement system changes that reliance on he RCT there will likely lead to false negatives.

While I can agree with a reasonably high degree of confidence that the former position makes sense (i.e. lets don't do any more homeopathy RCTs ) I have no operatinal knowledge about the value or reliability or weaknesses of the other assessment techniques that Berwick advocates. However, I can buy ( and have tried to sell) the notion that the RCT should not trump everything forever and always in assessing evidence in health care matters.Basic science implausibility may trump it at times-as in the woo arena. At others times, basic scientific principles and much everyday experience and common sense should prevail-as in the now overdone example of the parachute trial.