The Limits of Risk Factor Epidemiology

First a great quote, from Michale Thun, VP of Epidemiology and Surveillance Research at the American Cancer society:

With epidemiology you can tell a little thing from a big thing.What's very hard to do is to tell a little thing from nothing at all.

Gary Taubes in his widely cited article,"Epidemiology Faces Its Limits",Science, Vol 269,p. 164,July 1995, followed that quote with this comment:

...journals today are full of studies suggesting that a little risk is not nothing at all.

There is no basic law of science or statistics or epidemiology or metaphysics which will define how large a relative risk or a odds ratio has to be before physicians and patients need be concerned. Here we are talking about the interpretation or weighing of the evidence that accrues in the quest for evidence on which to base medicine.

Committees that author guidelines typically outline for the reader what their evidentiary hierarchy will be, usually randomized clinical trials at the top etc. But what are the rules for judging individual studies particularly the observational ones, e.g. case control studies and cohort studies.How big should a RR (or ORs for case-control studies) be before they consider that study worthy of adding to the pile of evidence worth of consideration ?

To get a sense of what the professionals do in that regard we could survey experts and see rules of thumb they use in knowing when to consider an observational study worthy of worrying about or suitable for publication. What we learn is that it may not be just the size of the RR but the overall context.

Robert Temple of the FDA is quoted by Taubes as saying:

My basic rule is if the relative risk isn't at least 3 or 4, forget it.

However, Dr. John Bailar,from McGill,believes there is no magic dividing line.

If it's a 1.5 relative risk and it's only one study and even a very good one, you scratch your chin and say maybe.

It is not size of the RR alone ( but we have to agree at some point low is too low say 1.03 relative risk) but the results of other studies addressing the same issue and concerns about scientific in general and in specific biological plausibility have to be factored in. Even though the size of the RR or OR ( odds ratio) is not necessarily determinative it is easy to cite a number of experts in the field who favor the notion that RR less than 2 should be- if not dismissed- at least looked at with a very skeptical eye.

While size of the relative risk is not the end of the analysis, it is the case that small RRs are more likely to be generated by undetected systematic error(s) than are large one. A RR of 1.2 should be much more suspect than a RR of 3.2

The observational studies can be considered coarse-grained instruments with bias and confounding being the basis for the coarseness,the hidden variables that can lead to an association that is not real. While calculating a confidence interval takes random variation in the data into account , bias and confounding lie outside its reach.

Sophisticated (and to many medical readers-mysterious) statistical methods such as various types of mathematical modeling may serve to eliminate or minimize some of the systematic errors but in the final analysis the reader or researcher still does not know to what extent biases are not controlled. I have yet to read the discussion section of an observational study in which the authors did not believe that they had "controlled" for sources of bias and confounding even when that study contradicted an earlier one whose authors also believed their methods likely excluded bias.

I believe that the best an ordinary medical reader -one whose wall is not decorated with an advanced degree in epidemiology or statistics-can do is:

1.Be very skeptical of Relative Risks less than 2 and particularly less than 1.5
2.Look in the articles' discussion sections for citation of confirming or contradictory studies.
3.Consider whether the findings fit some concept of reasonable biological plausibility.

Proton pump inhibitors, hip fracture and small relative risks

A JAMA article has generated considerable media attention citing a case control study that linked PPIs (proton pump inhibitors) to an increased risk of hip fracture.

The overall adjusted odds ratio was 1.44 with a confidence interval of 1.3-1.59. There was a dose response relationship demonstrated with duration of therapy.

Observational studies, such as this case-control study, are generally accepted to be so-called "hypothesis generating" studies rather than studies that more suggest a causative relationship (o.k. I'll stipulate causation is a deep, complicated philosophical issue and I use the word here just to indicate that case-control studies are just the beginning of efforts to demonstrate likely causal relationships).

A dramatic example of how observational trials can be misleading can be found in the juxtaposition of two studies in the New England Journal of Medicine dealing with relationship between hormone replacement therapy and coronary artery disease. One study demonstrated a two fold increase in coronary disease risk while the companion article showed a coronary artery risk reduction of 50% in those who used HRT.

There has been a number of conflicting case control studies regarding the relationship between statin use and colon cancer risk which again points our the problems involved when you take the results of one such study too seriously.

Let's quickly apply Bradford Hill's criteria to the PPI-hip fracture issue. The temporal relationship is there. There is a dose response relationship is a reasonable biological plausibility- low stomach acid conditions can interfere with absorption of some forms of calcium.
However, the association (1.44. OR) is not very large and there is not a strong consistency of results in other observational studies. (The JAMA article quote another study with similar results and one that did not demonstrate the association). At the end of the mini-analysis we still don't know if the reported relationship is valid or really know if we should somehow change our practice in regard to PPIs.

How big should the relative risk (RR) or odds ratio (OR) be for us to be concerned with the results? This is the very question asked by Dr. David Sackett in his book " Evidence-Based Medicine. How to practice and Teach EBM", Churchill Livingtone. Second edition,pg 162).For an answer to this the pioneers in EBM turn to argument by authority as opposed to an evidence based reason.

In regard to a case control study he said

We might not want to label an odds ratio from a case-control study as impressive unless it is greater than 4 for minor adverse events and set this value progressively lower [for more serious events].

Further, he says in regard to a cohort study, where there is less potential bias , that a relative risk of greater than 3 might be convincing for a more serious adverse event.I have posted before on the issue of RR in the 1-2 range.

John Ioannidis stirred up the medical community a bit in his article "Why Most Published Research Findings are False" and in regard to small relative risk determinations he is quoted as saying:

The smaller the effect sizes in a scientific field, the less likely the research findings are to be true...more likely true in scientific fields with ...relative risks [ in the 3- 20 range] ...than in scientific fields where the postulated effects are small [in the 1.1-1.5 range]

Could this study, and so many more with RRs less than 2 hit that hit the headline news, not really be worth much concern at all since the reported increased risk is so small ?

Dr. Marcia Angell, NEJM editor, has been quoted as saying that generally they only accept papers if the relative risk is 3 or more, particularly if it is biologically implausible or if it's a brand new finding. In the PPI case, there is some biological plausibility and occurs in the context of at least one study with findings pointing in the same direction.

In some legal venues, a relative risk of 2 or more is required to meet the legal standard of "more likely than not".

The PPI study authors suggest we should recommend that calcium supplements be taken with meals and I would add recommend adequate amounts of vitamin D ( thought to be about 800-1000 units per day now) and to perhaps have a looser trigger finger on the indications for measurement of bone density in those patients who are on long term PPI therapy and/or higher dose. However, I do not think patients whose GERD symptoms have been controlled by PPIs-and sometimes this change has been dramatic-should throw away their pills.

Is being a hospitalist the only way to still be a old time internist?

Dr. Laurence Wellikson seems to believe that the only way -at least a way-to recapture the core of what internist used to do (care for the patient with the complex problem(s))is to become a hospitalist.

Here is a point-counterpoint presentation by Dr. Willikson,CEO of the Society of Hospital Medicine,and Dr. Robert Centor,President of the Society of General Internal Medicine on the topic of "What is an Internist?"

Willikson seems right on target when he speaks about "internists [having] devolved into gatekepers and primary care physicians" ...[and] competing with Family Practitioners and Nurse Practitioners to be the traffic cop for resource use and burgeoning specialization."

He then inserts some of the current- in- vogue- jargon of the business consultants and speaks about the importance of preparing a "value proposition" and then makes what I consider to be a "boil the oceans" type suggestion namely to "reset the reimbursement system".

Given the current power relationship between third party payers and physicians it hard to conceive of what doctors could do to change the system. Hospitalists are either employees of the hospitals in which they practice or are contractors. The only tool either group has to bring about change is the strike, an action that physicians have rarely participated in. Changing the current medical care-reimbursement system is going to take more than hospitalists or internists writing a value statement, which I understand to be a statement made by a business explaining what they do and why the targeted customer should by it. (It occurs to me what if other specialists were exhorted to author a value statement.Perhaps neurosurgeons would propose "We take out brain tumors better than anybody."

Hospitalists do seem to spend most of their work day doing what many of us wanted to do when we decided to become internists and I hate to think the only avenue for the Oslerian type internist to travel is that of the hospitalist but that may be the reality though we have not quite arrived at that point yet. I hate to think that because an important part of what I used to do was not only caring for the complicated case in the hospital but also as an outpatient. This was a common occurence with COPD patients with exacerbations requiring hospital and often ICU type care. Internists should be the ones caring for patients with complex medical problems in and out of a hospital setting. There are still internists who care for the complex cases as in and out patients but clearly it is getting harder to do that. Dr. Wellikson seems to think internists should care for complex cases outside the hospital as well but leaves me wondering how he thinks that will happen.

Sowell's "Conflict of Visions" and medicine

Thomas Sowell's book "Conflict of Visions" is one of the most insightful and intellectually satisfying books I have ever read.It deals with the ways in which people differ in their fundamental view of the world.

Sowell observes that "reality is far too complex to be comprehended by any given mind" and therefore humans need maps, or a sense of how the world works to enable them to make sense of it all; he describes these maps as visions or pre-analytic cognitive constructs.

These visions differ about the basic nature of man. Sowell considers that this visions can be categorized into two broad categories are the "constrained view" also called the Tragic view and the "unconstrained view" also called the Utopian view.

In the Tragic view, man is constrained by his moral limitations and his egocentricity. The social challenge is how to work with those limitations rather than engage in energy wasting and ultimately futile efforts to change human nature.In this view, one deals with trade-offs and not solutions. Benefits to society derive from actions largely unintended but emerging from market actions which derive from the pressures of the incentives for individual gain which include the monetary and the psychic. Much of the meat of the constrained view can be picked from Adam Smith's "The Wealth of Nations". In a scheme that values trade offs , prudence assumes a high position. Burke said "Nothing is good but in proportion and with reference."

Steven Pinker in his book, 'The Blank slate' expresses the differences between the visions in this way :

In the tragic vision, humans are inherently limited in knowledge,wisdom and virtue and all social arrangements must acknowledge those limits....in the utopian, psychological limitations are artifacts that come from our social arrangements and we should not allow them to restrict our gaze from what is possible in a better world."

In the Utopian view, social inequalities such poverty,uninsured patients,drug addiction, racial or gender imbalances are seen as resulting from faults in society and therefore warrant that social targets be developed and plans made to rectify the social problems. The tragic Visionist would be concerned with the self interested motive of the people who are tasked with carrying out the programs and also with the unforeseen and unintended consequences that such projects seem to invariably entail.

Uneven distribution of wealth as brought about by a market economy in the unconstrained view is thought of as being unjust and in need of remedy in the name of social justice. The constrained view would argue that the notion of justice does not make sense when applied to an abstraction as society and would speak of justice only in the context of human decisions in a framework of laws.

So what has all of this to do with medicine? I believe the current iteration of Medical Professionalism ( Medical Professionalism in the New Millenium.A Physician Charter.Ann. Int. Med 5 Feb 2002, Vol. 136, pg 243-246) reflects the Utopian View and to a degree I find distressing has become part of the medical education agenda and the writing and speech of academic and organizational physicians. In this new charter, justice is emphasized and the justice proposed in their construct of social justice involves concern about "allocation of medical resources" and physicians considered stewards of society's resources.It talks about "commitment to a just distribution of finite resources" and to be committed to "develop guidelines for cost effective care".

Of course, concern for the individual patient's welfare has not be abolished and the "primacy of the patient welfare " is still said to be fundamental. However, a potentially conflicting principle,the social justice imperative, has been added the physician's obligation list.

I grew up medically with notion of concern for the individual patient being primary-first do no harm and to act always in the interest of the patient. Fiduciary duty was the decision trump card. I wonder at what point the imperative of "just allocation of resources" trumps concern for the welfare of the individual patient. The front line physician battered back and forth in the moment to moment flurry of phone calls, lab tests, patient problems, managed care restraints and hassels will have little time, energy or the tools to somehow factor into his patient decisions concern about "society's resources".

Now American College of Surgeons favor P4P-say it isn't so ,Joe.

Seemingly following in the footsteps of the American College of Physicians (ACP) the executive director of the American College of Surgeons (ACS) has written a letter to the NY Times advocating P4P. Dr. Thomas Russell wrote in part:

"Many physicians have long encouraged efforts to advance evidence-based care such as Medicare's "pay for performance" system. They do so because such steps are in the best interest of patients."

His comments seem mainly to be gratuitous asssertions which can be countered by equally gratuitous denials since he offers no arguments in support.Those of us who have serious questions about P4P and believe it to be simply wrong on many levels believe that is not in the interests of patients and that it will not advance evidence-based care.I have been of the opinion that the majority of real life internists in private practice do not share many of the policy decisions of ACP, I would like to hear from the prolific and articulate surgeons bloggers if a similar situation exists in the surgical world. I suspect it does.

More accusations of big pharma behaving very badly

The ethics of business is not the same as the ethics of medicine but if recent accusations against Lilly are true you might conclude that in regard to some businesses the word does not even apply. I have quoted Dr. Patricia Illingworth before:

...some commonly accepted principles of business ethics are fundamentally incompatible with traditional medical ethics...,

These practices include spinning and playing down the negative aspects of your product and intentionally misleading customers and maybe that is why we have laws against deceptive trade practices.

Reuters today published an article saying that the NY Times web site describes Lilly's activities promoting off label use of Zyprexa (olanzapine) and acting to play down Zyprexa's known side effects of significant weight gain and increased risk of diabetes. They are accused of a stealth campaign to promote the drug's use in dementia. Olanzapine has FDA approval for use only in Bipolar disease and schizophrenia.

NY Times writes about internal Lilly documents that suggest Zyprexa be promoted to primary care docs for use in dementia noting that psychiatrists are the physicians who mainly treat schizophrenia and bipolar disorder and primary docs do not feel confident is managing those conditions but they do manage many elderly patients with varying degrees of dementia.

Evidence is presented that indicates sales reps did in fact promote the drug for the off label use with the result of a significant increase in sales and were given monetary incentives for those activities. The promotional program was known as "Viva Zyprexa". A representative for Lilly denies claims of promoting the drug use for non approved indications and is quoted as stating that a Zyprexa-diabetes link has not been proven.

FDA relegates Ketek to second line treatment for community acquired pneumonia

The FDA seems to making some progress in the Ketek matter.Maybe now we will be seeing less of the ubiquitous Ketex advertising owl on the back of medical journals.

I have been concerned with the Ketek issue for some time and have blogged on and on about it on several occasions.I expressed the opinion that there was little if any indication for that drug.It had been approved and marketed for sinusitis,bronchitis and community acquired pneumonia (CAP).Now the FDA panel says use it only as a second line drug for CAP.At least they are heading in the right direction. The risk of liver disease seems real (I believe at least 12 fatal cases so far) and why anyone would use it to treat sinusitis escapes me.

ADDENDUM:(12/19/06) Thanks to an alert reader I was informed I had misspelled the brand name of the drug in question when this message was first published.Since one who controls the present controls the past I am now going to at least partially control the past and republish the corrected posting with the explanatory addendum. James Gaulte

Post-modernism,EBM and George Orwell

Dr. Roy Poses in his Dec.01,2006 posting on Health Care Renewal tackled the topic of the attack of the post-modernists on Evidence Based Medicine (EBM). From time to time, I have written about the mis-use of EBM to cook the RCT books and hype the results and about how some may misunderstand EBM but the post-modernists attack on EBM seems to be basically self contradictory.

As pointed out by Steven Pinker (pg 426 ," The Blank Slate, the modern denial of human nature")

"It is ironic that a philosophy that prides itself on deconstructing the accoutrement's of power should embrace a relativism that makes challenges to power impossible, because it denies that there are objective benchmarks against which the deceptions of the powerful can be evaluated."

Pinker prefaces those remarks by discussing the totalitarian regime found in "Nineteen Eighty Four" and by describing their philosophy as "thoroughly postmodernist".

He quotes from the government agent, O' Brien, as he lectures the protagonist, Winston Smith, who is strapped to a table and tortured and "educated ".

"You believe that reality is something objective,external, existing in its own right. ...But I tell you, Winston, that reality is not external .[it exists] only in the mind of the party, which is collective and immortal"

When there is no objective truth and no means of obtaining the truth,whoever is in charge determines what it true or not. "Who controls the past controls the future:who controls the present controls the past."

A frontal attack on the science of medicine or on science in general is fairly easily beaten back.Nonsense tends to fall on its own weight even if sometimes that fall may take longer than we would like. In regard to medicine and medical education the greater danger seems to be in the infiltration of alternative medicine elements-and its philosophic underpinnings,Post Modernism- into the curriculum of medical schools and even into the clinical services they offer. Dr. RW relentlessly keeps us well informed about the alarming instances of medical schools offering courses and sponsoring clinics for practitioners of the various categories of altie hokum.

Doctors fight back, Insurance company capitulates

The above headline was inspired by a news report in the daily email I get from AMA ( one of the few perks of membership). I had blogged previously about the Blue Shield matter in Seattle.

Apparently, something call the "AMA Litigation Center" joined in a legal action with the Washington State Medical Society and six physicians against the "Select Network Plan" devised by Regence Blueshield. I had not heard of the Litigation Center, but it seems that for the past 10 years it has been involved in legal actions that look after the interests of physicians and has a new initiative to fight back against insurers who play fast and loose with bogus guidelines and performance indicators "delisting" docs and damaging their reputations and practices.

The insurer dropped the plan.

Dr. Pleasted, president of AMA, is quoted as saying:

"A patient's choice of a physician must not be influenced by a health insurer's mistaken assumption that low cost is the only acceptable measure of quality."

Physicians should not neglect the powerful effect that legal action that sometimes have but what happens now is not clear.What about the patients that received letters from the insurance company informing them that their doctors were not practicing according to some standards? They were said to not be practicing "quality and efficient " medicine. What reparations will be made for lost patients,damaged reputations and lost income? According to AMA websites ,( I believe membership is required to access) the law suit continues in an effort to obtain monetary damages.

JAMA article considers Hospital Performance Measures and mortality rates

In the December 13, 2006 issue of JAMA, Drs. Rachel Werner and Eric Bradlow analyzed data from 3657 acute care hospitals looking at the relationship between certain performance measures and mortality rates. They used 10 measures which are available from CMS and the Joint Commission 's web site. For example, in regard to pneumonia the measures are: 1) timing of initial antibiotic administration, 2) pneumococcal vaccination and 3)whether oxygenation was measured in the first 24 hours.

The authors concluded that:

"hospital performance measures predict small differences in hospital risk-adjusted mortality rates"
and said further:

"Based on these results, the ability of performance measures to detect clinically meaningful differences in quality across hospitals is questionable"

Dr. Susan Horn in her related editorial makes several good points:

1.If these performance measures are not strongly associated with outcomes, why should we bother with them either as basis for P4P of for consumers to use as tools for judging hospitals?
2.Since many( but not all) of these measures were chosen,at least,in part because of the results of RCTs, why are these interventions not associated with better outcomes in practice?

Her answer to the second question is that improving outcomes in actual real world practice is much more complex and multidimensional than using a few, discrete interventions that seemed efficacious and safe in RCTs that typically focus on a single condition in a fairly homogeneous population. In part, it is the question of efficacy versus effectiveness. Measuring quality of care is much more complex and slippery than the ten measures analyzed in their study.

In an earlier article Dr. Werner pointed out some possible unintended consequences of hospital "report card" including treating the chart and excluding sicker patients.

One message should be: we have no business using simple and simplistic measures as a basis for pay for performance or for claiming to be able to distinguish between different hospitals quality of care. Not only may they not deliver on what they promise, they may be harmful.

AMA President nails P4P for what it is at AMA Interim meeting

Dr. William G. Plested III, president of the AMA had this to say regarding pay for performance at the recent Interim meeeting;

I will point out that-reminiscent of the managed care debachle-P4P will allow insurers to dictate the treatment that we give our patients and will publicly label any physician foolish enough to contract with them and not follow their dictates as nonpreferred, substandard or some such label.

This is not just speculation as the physicians in the state of Washington had exactly that happen to them.

Further, he said that he was unaware of any P4P program in place that was in compliance with the AMA suggested guidelines for P4P.

A proposal was made for the AMA to launch a campaign to discredit P4P and correctly label it as economic credentialing. Unfortunately, no resolution was passed. At least, the president of the AMA recognizes P4P for what it is which much more than can be said for the leadership of the American College of Physicians who seem to be working with the third payers payers to move ahead with this very bad idea. ACP seems to believe-or at least their rhetoric suggests-that they can "work with" third payers to ensure that P4P program will improve quality and not just lower costs.

Randomized trials are wonderful but beware of "evidence based paralysis"

The term "evidence based paralysis" came to my attention in a letter to the editor in the Archives of Internal Medicine in the Aug 14/28 2006 Letter to the Editor section. Drs. David Ziemer and Lawrence S. Phillips from Emory used it in their reply to a comment about an article concerning the issue of whether tight control of type 2 diabetic patients is " evidence based ". They also used to term "RCTomyopia" to refer to an " unwillingness to take action without incontrovertible proof from controlled trials".

RCTs and meta-analyses are considered the most reliable tools in the hierarchy of EBM.(For some time I have questioned whether MA belong in that position.) Because of this, some make the mistake of jumping to the position that if there are no RCTs that there is no evidence based reason for action.

Of course, RCTs are good, but there are not good for everything.They cannot answer all the questions physicians need/want to have answered.

They are great for discrete interventions in carefully defined, relatively homogeneous conditions in terms of determining efficacy. They are clearly less good for determining harm and for determining how to diagnosis conditions and for determining prognosis. In regard to harm, RCTs can recognize relatively common adverse effects that occur fairly soon after a medication is started but are less useful in detecting less common and/or delayed side effects.

However, when the questions to be answered arise in and from more complex patient populations in which the patient characteristics and interventions are more complex and heterogeneous it become much more difficult to separate out causality from bias,confounding and even random variation.In fact, some times it is more than difficult in that after the trial has been done and analyzed we still do not know the answer we were searching for. Case in point is the recent back surgery for herniated disc versus conservative management RCT published in JAMA. In this instance, there was so much cross over between the two groups as they were randomized that the "intent to treat" analysis was not considered valid and the "as treated analysis" suffers from the very real risk of selection bias which is why we have randomization in the first place. One is left with the disturbing thought that it may not be possible to solve this clinical issue by doing a randomized trial as long as we deal with patients who are free to do what they think is best for them.

It is important to know the limitations of the RCTs and that the very nature of certain clinical problems may be too complex for RCTs to be of use. Further, RCTs can never be done for all the problems that they are suitable to analyze.There are too many questions, often too little money and interest and with new drugs, procedures and testing methods always evolving-or at least changing-older RCTs lose relevance and the newer treatments may be years away from RCT results.

In treating patients with serious illnesses often we have to act not just sit there. We have to go with the evidence we have not the evidence we would like to have.

Ziemer and Philips had this to say in their letter to editor:

We believe that responsible physicians and patients should make decisions based on the best available evidence-including cell and animal studies, observational studies and controlled trials if available-and the strengths and weaknesses of the findings with each approach should be given due consideration.

I cannot resist when discussing this general topic to throw in the parachute-gravitational challenge comment that I quoted last year.