Table of Contents:

  Health care reform, information overload and the role of  effectiveness evidence

No one would doubt that world’s health care system is in process of significant change because now the majority of the world population is being served by some form of managed care organization. It can be argued that this change is being largely, if not exclusively, driven by efforts to hold down the costs of care and, some might add, to bolster the bottom line of for-profit health care systems.

Issues of quality of care are currently coming to the forefront of the debate. An appropriate question is, “What role does effectiveness evidence play in third-party reimbursement decisions for approved, covered, and reimbursed services?” If rational decision making were indeed based on evidence, then a serious problems would be identifying the best evidence and distinguishing information from evidence. Dictionaries define information as news, whereas evidence is something that tends to prove or provide grounds for holding a certain belief. A higher standard exists for something to be called evidence rather than simply information – yet this distinction is often lost.

A recent editorial in the British Medical Journal bemoaned, “We have much too information of poor quality, too little that’s good, and no effective way of sorting it out” [Slide1]. Advances in medical science and information technology are combining to create ever-increasing amounts of information, resulting in what has been commonly termed information explosion at a systems level, creating the challenge of information overload at the individual level. This phenomenon is evidenced by a more than 10-fold increase in the number of professional journals across all disciplines. Mulrow estimates that approximately 2 million medically related articles are published annually [Slide2]. If you also use electronic mail and access the Internet regularly; you may feel like the patient suffering from information overload. [Figure1]

To make matters worse, research studies designed to answer the same question do not always reach consistent conclusion. In some cases, studies reach conflicting conclusions, leaving people confused as to what the “right” course of action is. [Figure2]. This leads to a general skepticism about the worth and credibility of the scientific enterprise. The random medical news from the New England Journal of Panic Inducing “Gobbleygook” randomly assigns cause, effect (i.e., resulting medical condition or outcome), and population (e.g., adults, children, men, women) to studies.

  Finding high quality evidence in the information age

So what is one to do? A possible solution is to turn to one of the available decision support systems. These include (1) literature databases like MEDLINE or its European counterpart, EMBASE, (2) computer-based textbooks that offer enhanced (Boolean) searching capabilities for identifying relevant information, (3) drug databases that alert users to the potential side effects and drug interaction, (4) clinical practice guidelines, (5) reminder or alert systems, (6) diagnostic or treatment decision support systems that provide consultations once the characteristics of a patient’s problem are entered in the system, and (7) systematic reviews of the effectiveness of health care interventions, offers some unique advantages or “added value” over the others, and I describe one particular database of systematic reviews (the Cochrane Database of Systematic Reviews [Slide3]) that may be particularly useful for the physician, patient, and for policy-oriented decision making.

Searching MEDLINE, EMBASE, or other health-related databases for articles that are relevant to a particular patient’s problem often results in many references. These references typically describe studies whose research designs, patient groups, methods, and even results often differ. It is not easy to make sense of and synthesize these varied findings into a concrete summary of the evidence. This is exactly the goal of a good systematic review (or meta-analysis), which can also inform about the consistency and replicability of the findings across studies. Electronic textbooks, such as Harrison’s Textbook of Medicine or Scientific American Medicine, both of which are available on CD-ROM, offer the advantage of being easier to search than standard, printed texts. Electronic drug databases also can be useful reference and decision support tools.

Information in textbooks and drug databases, however, must be based on good evidence, which a well-done, systematic review or meta-analysis can provide good source of evidence required to construct practice guidelines, which typically rely on the best available evidence and/or expert opinion as the basis for making recommendations. Some of the best guideline sources can be found in the Canadian and U.S. Prevention Task Force Guidelines, as well as in the National Guideline Clearinghouse.

Systematic reviews also can be good sources of the evidence that is necessary to develop decision rules for treatment or diagnostic computer consultation programs (e.g., DXplain, ILIAD, or QMR) [Slide4].These decision support tools allow clinicians to enter their patient’s findings into the system to receive a differential diagnosis or treatment plans ranked on the basis of likelihood of accuracy or effectiveness, respectively. What is a systematic review, then [Slide5], [Slide6]?

Elements of high quality systematic reviews of effectiveness evidence

According to the Cochrane Collaboration, a good systematic review has at least eight key elements.

Adapted from: Chalmes I. The Cochrane Collaboration: preparing, maintaining and disseminating systematic reviews of the effects of health care [review]. Ann NY Acad Sci 1993; 703: 156-163.

The first is to state the objectives of the review and outline the eligibility criteria for including and excluding studies in the review. This is typically not done in more traditional narrative reviews and is a key element in trying to elevate the quality of a review by enhancing its “objectivity”. Explicit criteria for inclusion and exclusion of studies in the review should specify the population, intervention, outcome, and methodologic criteria for the studies included in the review.

The next steps are to search for all the studies, both published and unpublished, that seem to meet eligibility criteria and tabulate the relevant characteristics of each study. Comprehensive search methods are used to locate relevant studies by examining reference lists of key articles (often referred to as the ancestry approach) and conference proceedings, directly contacting experts in the field, and searching a wide range of computerized databases using a combination of appropriate key words and free text searching strategies. Coding the characteristics of each study typically includes an assessment of the methodologic quality of the study using predefined criteria.  

Coding should attempt to avoid bias and be reproducible, which typically requires each study to be coded independently by at least two “judges”. Exploration of variation between the findings of the studies enables the authors and readers to determine the degree to which characteristics of the study design itself may be accounting for some of the findings.  Even differences in the design of randomized controlled trials have been shown to account for differences in the effect of therapy. This is particularly true for the precise manner in which patients are assigned to the various treatment groups. The coding of study quality essentially serves as an assessment of the validity of the primary studies.

The next step is to apply the eligibility criteria to each study and to justify the exclusion of each study. A table listing the excluded studies, with the reasons justifying their exclusion is typically included as part of a systematic review, along with a table summarizing the characteristics of the studies included in the review. This should result in the most complete data set feasible, often including information to supplement published data that is requested and obtained directly from the authors of the primary studies.

The results from the eligible studies are then synthesized and summarized. If appropriate and possible, results from each of the primary studies are pooled quantitatively using statistical synthesis methods (i.e., meta-analysis). Regardless of whether results are pooled quantitatively, it is important to adhere to the other elements of a systematic review to maintain as much objectivity and minimize as much bias in the review as possible, It is important to recognize that a systematic review is not a meta-analysis if it does not quantitatively pool results across studies. Conversely, a meta-analysis is not a systematic review if it does not explicitly state (1) the eligibility criteria for including studies in the review, (2) what databases have been searched, and (3) what search terms were used to identify relevant studies for the review.

Cochrane Collaboration and Cochrane Database of Systematic Reviews

Evidence obtained from at least one good, properly designed randomized clinical trial traditionally has been considered the highest quality evidence available [Slide7]. Evidence from a properly designed systematic review of all the available properly design randomized trials has been deemed to be evidence of the highest quality from use in decision making, and it is superior to the results from any one randomized clinical trial. The Cochrane Database of Systematic Reviews [Slide8] is an electronic journal published quarterly that first appeared in 1995. The Cochrane Database of systematic reviews is one of four components of what has evolved into the Cochrane Library. The other three components are (1) the Database of Abstracts of Review of Effectiveness (DARE), (2) the Cochrane Controlled Trials Register, and (3) the Cochrane Review Methodology Database.  

Unlike articles appearing in medical journals, for example the New England Journal of Medicine or Journal of the American Medical Association, one can be interpreted any review in the Cochrane Database of Systematic reviews regardless of subject matter or areas of medicine if one is able to interpret the logo of the Cochrane Collaboration. Each of the seven horizontal lines represents the 95% confidence interval for the results of one randomized trial comparing the effectiveness of an intervention with standard care or placebo,

More certain results are represented by shorter lines, with horizontal lines crossing the vertical line indicates that the study result favors standards therapy or placebo. A horizontal line to the left of the vertical line indicates that the study result favored the intervention, whereas a horizontal line to the right of the vertical line indicates that the study result favors the therapy or placebo. For example, the top trial favors the intervention, whereas the two below it suggest an advantage in favor of the intervention, but seem to border on “no difference”, whereas the fourth trial from the top clearly indicates that the two treatments have similar effects. The pooled result across all seven trials is summarized by the diamond.

 Results from any systematic review in the Cochrane Database of Systematic Reviews are interpreted in an analogous fashion, regardless the topic of the review [Slide9].

Figure 3. summarizes an interesting subgroup analysis based on the length of time of the intervention. The authors of this review were able to locate only two randomized trials in which the intervention lasted no more than 6 months. These two trials involved a total of 167 patients and showed the interventions to be equally effective in preventing relapse. Thirteen of 86 patients in the intervention group experienced relapse, whereas 14 of 81 patients receiving standard care experienced relapse. On the other hand, for interventions lasting between 7 months and 1 year, the results for all patients pooled across all seven trial showed a significant reduction in relapse for patients in the family intervention groups (105 of 313 relapsed) compared with standard care (127 of 252 patients relapsed). This resulted in an average reduction of 58% in the odds of relapse – that is, 1 minus 0.42 under the column labeled Peto OR. At this point it is important to review the language used to report and discuss the results of research, including such terms as odds, risks, and survival.

Applying evidence to individual patients

So the question remains, what is the best way to communicate the results of research studies to make the most informed decisions? Decisions based on a reduction in risk of 34% tend to vary from decision based on an absolute reduction of risk of less than 1%, although both measures are derived from the same evidence. Empirical data examining the impact of communicating information on actual behavior are lacking, as research in this area is in the early stages of development. Perhaps one way to dealing with this problem when discussing options with patients, particularly those who desire to be more involved in decision regarding their own care, would be to take the time to walk them through the various representations of the evidence so that they may be fully informed, particularly when the consequences of the decisions are greater.

It is not easy to apply evidence-based medicine in everyday clinical practice. Haynes has identified at least four major barriers. These include: (1) the difficulty in finding sound evidence, (2) the lack of clarity of standards of evidence and interpretation, and (3) the mismatch between existing evidence and the clinical situation at hand. All three of these barriers are affected by a huge fourth barrier that underlies the first three: the increasing time pressures and constrains on clinicians that make it difficult, if not almost impossible, to find, to synthesize, and interpret evidence in real time that is relevant to a patient’s problems. Many experiments both planned and naturally occurring are under way to try to overcome these barriers. Just as pharmacists, physician assistants, social workers, and other health professionals are becoming part of group physician practices, so too might information specialist, to whom physicians may turn for identifying evidence. Just as patient care and management plans are reaffirmed or modified when the results of laboratory tests arrive after the patient goes home, so too might they be reaffirmed or modified after the evidence becomes available.

We should keep in mind that evidence-based medicine is not a panacea whose practice will lead to the best solution to each patient’s problem. Evidence-based medicine involves integrating clinical expertise with the best available clinical evidence derived from systematic research, along with patient preferences and choice. It is not intended to replace clinical experience and judgment but rather inform and enhance it. The practice of evidence-based medicine is a process of lifelong, self-directed learning in which caring for patients creates a need for clinically important information about diagnoses, prognoses, treatment, and other health care issues.

Straus and Sackett assert that practicing evidence-based medicine is a way for clinicians to keep up with the rapidly growing body of medical literature and to sharpen their skills in asking answerable questions and finding the best evidence to answer these questions by providing a framework for critically appraising evidence. The goal is to integrate valid and useful evidence with clinical expertise and each patient’s unique feature to provide the best care possible [Slide10]. The steps that are considered to effectively practice evidence-based medicine are shown in table 3.

Resource: SE Straus, DL Sackett. Using research findings in clinical practice. BMJ 1998; 317:339-342

It is critical always to evaluate how well your decisions and actions based on the evidence actually work when applied to the patient and to modify future decisions and actions accordingly in what may be a very “trial and error” iterative process for some patients.

Some of the factors that are helpful to consider when applying evidence to individual patients have been described by Sheldon et.al. and are summarized in table 4.

Resource: TA Sheldom, GH Guyatt, A Haines. Getting research findings into practice: when to act on the evidence . BMJ 1998; 317:339-342

Sheldon summarize these steps well when they state, “the decision whether to implement research evidence depends on the quality of the research, the degree of uncertainty of the findings, the relevance to the clinical setting, whether the benefits to the patient outweigh any adverse effects, and whether the overall benefits justify the costs when competing priorities and available resources are taken into account”. They emphasize that “systematic reviews that show consistent results are likely to provide more reliable research evidence than non systematic reviews or single studies”. Many social, organizational or institutional barriers exist that inhibit the incorporation of research-based evidence into clinical practice. The fact it is often difficult to locate and access patient-relevant information as the point of care should not stop us from using the best evidence that does exist in our decision making. The efforts of individuals working within the Cochrane Collaboration and elsewhere, combined with the ongoing remarkable advances in information technology, virtually ensure that the amount and accessibility of valid, reliable evidence to use for informed decision making continues to grow and become increasingly available in real patient care time. It is incumbent on us that we begin to make the best use of the best evidence that we can. [Slide11]