More important caveats re: generalizing from Randomized trials

In the Jan. 9, 2007 issue of the Archives of Internal Medicine, there is an interesting article regarding the external validity of randomized clinical trials. External validity refers to the ability of results of a trial or experiment to generalize to the real world population.

There has been much written about how often the elderly (usually 65 and over) and women are excluded from trials. Clinician have been frequently admonished to be careful about extrapolating results of trials to their patients who differ from the those eligible to have participated in the particular trial. This article raises interesting questions suggesting that those patients who apparently did not differ and were eligible but did not participate in the trial had a higher baseline risk and worse results. Of course, those who were not eligible had a even higher baseline risk and outcome.

Dr.Steg and a group of international investigators divided patients in the GRACE registry (which forms a large multi-national cohort,) into three groups,RCT participants, eligible patients not enrolled in a trial, and ineligible patients. Not only was there a gradient of baseline risk with those participating in RCTs having the lowest, those eligible but not participatng the next lowest and those not eligible the highest but an important observation was that the RCT participants had half the hospital mortality of the eligible, nonparticipating group. (3.6% versus 7.1%)

The authors offered several possible interpretations.The one that strikes my eye is that the better results of the trial patients is due to the "closer medical attention" possibly provided in the trial,e.g perhaps more regular use of the other treatment modalities-aspirin, beta-blockers, reperfusion therapies). Apparently, they had some evidence that this was a factor.In addition, as always, the specter of "unknown confounders" is raised which in statistic-speak means "it could be something else but we don't know what".

What are the implications of this? One is that the treatment groups outcomes would be better irrespective of the benefit derived from the particular therapy at issue.The results of thrombolysis in S-T segment elevated M.I.,for example, may be less impressive even in those patients who characteristics closely resemble those of the patients treated in the trial.

The authors seem to provide evidence for still another reason for the effectiveness-efficacy gap. Even if the direction of the results of a RCT may be correct, we should not expect the results to be as impressive when the therapies are applied to patients even if their clinical characteristics closely mirror those who participated in the trial.