Potential Strengths and Limitations of a Systematic Review
Systematic Review and Meta-analysis: When I Study Is Merely not Enough
We live in the information age, and the practise of medicine is becoming increasingly specialized. In the biomedical literature, the number of published studies has dramatically increased: At that place are at present more than 15 million citations in MEDLINE, with ten,000 to 20,000 new citations added each calendar week (1). Multiple relevant studies usually guide well-nigh clinical decisions. These studies oftentimes vary in their design; methodologic quality; population studied; and the intervention, test, or condition considered. Considering fifty-fifty highly cited trials may be challenged or refuted over fourth dimension (2), clinical determination-making requires ongoing reconciliation of studies that provide different answers to the same question. Both clinicians and researchers tin besides benefit from a summary of where doubtfulness remains. Because it is often impractical for readers to track downwardly and review all of the main studies (three), review articles are an important source of summarized evidence on a particular topic (4).
Narrative Review, Systematic Review, and Meta-analysis
Review articles have traditionally taken the grade of a narrative review, whereby a content skillful writes most a particular field, condition, or treatment (5–7). Narrative reviews have many benefits, including a broad overview of relevant information tempered by years of practical knowledge from an experienced author. Indeed, this article itself is in a narrative format, from authors who take published a number of meta-analyses in previous years.
In some circumstances, a reader wants to become very knowledgeable nigh specific details of a topic and wants some assurance that the information presented is both comprehensive and unbiased. A narrative review typically uses an implicit process to compile evidence to support the statements being made. The reader often cannot tell which recommendations were based on the author's clinical experience, the latitude to which available literature was identified and compiled, and the reasons that some studies were given more than accent than others. It is sometimes uncertain whether the author of a narrative review selectively cited reports that reinforced his or her preconceived ideas or promoted specific views of a topic. Also, a quantitative summary of the literature is ofttimes absent in a narrative review.
A systematic review uses a process to identify comprehensively all studies for a specific focused question (drawn from research and other sources), assess the methods of the studies, summarize the results, present primal findings, place reasons for unlike results across studies, and cite limitations of electric current cognition (8,9). In a systematic review, all decisions used to compile data are meant to be explicit, assuasive the reader to approximate for him- or herself the quality of the review process and the potential for bias. In this way, systematic reviews tend to be more than transparent than their narrative cousins, although they too tin be biased if the selection or emphasis of certain chief studies is influenced past the preconceived notions of the authors or funding sources (10).
Depending on the nature of the data, the results of a systematic review tin can be summarized in text or graphic class. In graphic grade, it is common for different trials to be depicted in a plot where the point estimate and 95% confidence interval for each study are presented on an individual line (11). When results are mathematically combined (a process sometimes referred to as pooling), this is referred to as meta-assay. Graphically, the pooled issue is often presented as a diamond at the lesser of the plot.
When performing a meta-assay, a review team usually combines amass-level data reported in each main study (point and variance estimate of the summary mensurate). On occasion, a review team will obtain all of the private patient data from each of the primary studies (12,13). Although challenging to bear (14), individual patient meta-analyses may have certain advantages over amass-level analyses. As highlighted in a review of angiotensin-converting enzyme (ACE) inhibitors for nondiabetic kidney affliction, this includes the utilise of mutual definitions, coding and cutoff points between studies, addressing questions not examined in the original publication, and a ameliorate sense of the impact of individual patient (versus study level) characteristics (12,xv).
As starting time highlighted a decade ago (16), the number of systematic reviews in nephrology and other fields has increased dramatically with fourth dimension, paralleling the rapid growth of biomedical literature during the past half century. Initiatives such equally the Cochrane Collaboration accept further increased the profile and rigor of the systematic review process (details of the structured process of Cochrane systematic reviews are available through their Spider web site) (17,eighteen). From 1990 to 2005, there were more than 400 systematic reviews and meta-analyses published in the subject area of nephrology (Figure ane). Of these reviews, 40% pertained to chronic kidney illness or glomerulonephritis and 20, 16, fifteen, and 7% pertained to kidney transplantation, dialysis, astute kidney injury, and pediatric nephrology, respectively. As a publication type, however, systematic reviews accept not been without controversy: Some authors consider a meta-assay the all-time possible use of all available information, whereas others question whether they add anything meaningful to scientific knowledge (19). The strengths and weaknesses of this publication type are described side by side.
- Download effigy
- Open in new tab
- Download powerpoint
Figure ane.
There take been more than 400 systematic reviews and meta-analyses published in the field of study of nephrology since 1990, with the annual number increasing with time. Frequencies were estimated from a MEDLINE and EMBASE search performed by an experienced renal librarian in December 2006. Citations were reviewed by a nephrologist for relevance. Duplicate publications from the same group of authors were counted only once.
Strengths of Systematic Review and Meta-analysis
Physicians make better clinical decisions when they understand the circumstances and preferences of their patients and combine their personal experience with clinical evidence underlying the available options (20). The public too expects that their physicians volition integrate research findings into practice in a timely way (21). Thus, sound clinical or health policy decisions are facilitated by reviewing the bachelor testify (and its limitations), understanding reasons why some studies differ in their results (a finding sometimes referred to as heterogeneity among the chief studies), coming upward with an cess of the expected effect of an intervention or exposure (for questions of therapy or etiology), so integrating the new information with other relevant treatment, patient, and health care system factors.
In this respect, reading a properly conducted systematic review is an efficient way to go familiar with the best available research evidence for a focused clinical question. The review squad may besides have obtained information from the principal authors which was not available in the original reports. The presented summary allows the reader to take into account a whole range of relevant findings from research on a particular topic. The process can also found whether the scientific findings are consistent and generalizable across populations, settings, and handling variations and whether findings vary significantly past particular subgroups. Once again, the potential strength of a systematic review lies in the transparency of each phase of the synthesis process, assuasive the reader to focus on the claim of each decision made in compiling the data, rather than a simple dissimilarity of 1 written report to another as sometimes occurs in other types of reviews.
For example, studies demonstrating a significant effect of treatment are more probable to be published than studies with negative findings, are more likely to be published in English language, and more likely to be cited by others (22–27). A well-conducted systematic review attempts to reduce the possibility of bias in the method of identifying and selecting studies for review, past using a comprehensive search strategy and specifying inclusion criteria that ideally take not been influenced past a priori knowledge of the principal studies.
Mathematically combining data from a series of well-conducted primary studies may provide a more precise gauge of the underlying "true effect" than any private study (28). In other words, past combining the samples of the individual studies, the size of the "overall sample" is increased, enhancing the statistical power of the assay and reducing the size of the confidence interval for the point estimate of the effect. It is also more efficient to communicate a pooled summary than to draw the results for each of the individual studies. Sometimes, if the treatment consequence in modest trials shows a nonsignificant tendency toward efficacy, then pooling the results may establish the benefits of therapy (16). For instance, ten trials examined whether ACE inhibitors were more constructive than other antihypertensive agents for the prevention of nondiabetic kidney failure (29). Many of the 95% confidence intervals for the estimate provided by each written report overlapped with a finding of no effect; however, the overall pooled estimate established a benefit of ACE inhibitors.
For these reasons, a meta-analysis of similar, well-conducted, randomized, controlled trials has been considered one of the highest levels of evidence (30–32). It is of import to stress that the main trials all take to be conducted with loftier methodologic rigor for the meta-analysis to exist definitive. Alternatively, when the existing studies have important scientific and methodologic limitations, including smaller sized samples (which is more oftentimes the example), the systematic review may identify where gaps exist in the bachelor literature. In this instance, an exploratory meta-assay can provide a plausible approximate of effect that tin be tested in subsequent studies (33,34).
Limitations of Systematic Review and Meta-analysis
This type of publication type has many potential limitations that should be appreciated by all readers. Start, the summary provided in a systematic review and meta-analysis of the literature is simply equally reliable as the methods used to judge the consequence in each of the master studies. In other words, conducting a meta-analysis does non overcome problems that were inherent in the design and execution of the primary studies. Information technology also does not right biases as a result of selective publication, whereby studies that report dramatic effects are more likely to exist identified, summarized, and subsequently pooled in meta-assay than studies that study smaller issue sizes (an issue referred to equally publication bias). Because more than than three quarters of meta-analyses did non report whatsoever empirical assessment of publication bias (35), the true frequency of this form of bias is unknown.
Controversies besides arise around the interpretation of summarized results, peculiarly when the results of discordant studies are pooled in meta-assay (36). The review process inevitably identifies studies that are diverse in their design, methodologic quality, specific interventions used, and types of patients studied. At that place is often some subjectivity when deciding how similar studies must exist before pooling is appropriate. Combining studies of poor quality with those that were more than rigorously conducted may non be useful and can atomic number 82 to worse estimates of the underlying truth or a imitation sense of precision around the truth (36). A imitation sense of precision may also arise when various subgroups of patients defined by characteristics such every bit their age or gender differ in their observed response. In such cases, reporting an aggregate pooled effect might be misleading if there are important reasons to explain variable treatment effects across unlike types of patients (36–xl).
Finally, simply labeling a manuscript as a "systematic review" or "meta-assay" does not guarantee that the review was conducted or reported with due rigor (41). To reduce the chance of arriving at misleading conclusions, guidelines on the deport and reporting of systematic reviews were recently published (42,43); still, of import methodologic flaws of systematic reviews published in peer-reviewed journals have been well described (44–54). For example, of the 86 renal systematic reviews published in 2005, the majority (58%) had of import methodologic flaws (Mrkobrada G, Thiessen-Philbrook H, Haynes RB, Iansavichus AV, Rehman F, and Garg AX, submitted). The near common flaws amid these renal reviews were failure to assess the methodologic quality of included primary studies and failure to avoid bias in study inclusion (Mrkobrada 1000, Thiessen-Philbrook H, Haynes RB, Iansavichus AV, Rehman F, and Garg AX, submitted). In some cases, industry-supported reviews of drugs have had fewer reservations about methodologic limitations of the included trials than rigorously conducted Cochrane reviews on the same topic (10); yet, the hypothesis that less rigorous reviews more than ofttimes study positive conclusions than good-quality reviews of the same topic has not been borne out in empirical assessment (48,53,55). Nonetheless, similar all good consumers, users of systematic reviews should carefully consider the quality of the product and adhere to the dictum "caveat emptor": Let the heir-apparent beware. The limitations described in this section may explicate differences in the results of meta-analyses as compared with subsequent large, randomized, controlled trials, which have occurred in approximately one 3rd of cases (56).
How to Appraise Critically a Systematic Review and Meta-analysis
Users of systematic reviews demand to assure themselves that the underlying methods used to get together relevant information were audio. Before because the results or how the data could exist appropriately applied in patient care (9), there are a few questions that the reader can inquire him- or herself when assessing the methodologic quality of a systematic review (Table ane).
Tabular array i.
Questions to ask when assessing the quality of a systematic reviewa
Was the Review Conducted According to a Prespecified Protocol?
It is reassuring if a review was guided by a written protocol (prepared in advance) that describes the research question(s), hypotheses, review method, and plan for how the information will be extracted and compiled. Such an approach minimizes the likelihood that the results or the expectations of the reviewing squad influenced study inclusion or synthesis. Although almost systematic reviews are conducted in a retrospective manner, reviews and meta-analyses can in theory be defined at the time several similar trials are being planned or under way. This allows a set up of specific hypotheses, data collection procedures, and analytic strategies to be specified in advance before any of the results from the principal studies are known. Such a prospective effort may provide more than reliable answers to medically relevant questions than the traditional retrospective approach (41).
Was the Question Focused?
Clinical questions frequently bargain with issues of treatment, etiology, prognosis, and diagnosis. A well-formulated question normally specifies the patient'due south problem or diagnosis, the intervention or exposure of interest, any comparison group (if relevant), and the primary and secondary outcomes of interest (57).
Were the "Right" Types of Studies Eligible for the Review?
Different study designs can exist used to answer different clinical questions. Randomized, controlled trials; observational studies; and cross-sectional diagnostic studies may each be advisable depending on the primary question posed in the review. When examining the eligible criteria for study inclusion, the reader should feel confident that a potential bias in the selection of studies was avoided. Specifically, the reader should ask her- or himself whether the eligibility criteria for report inclusion were advisable for the question asked. Whether the correct types of studies were selected for the review also depends on the depth and breadth of the underlying literature search.
For case, some review teams will consider only studies that were published in English. There is evidence that journals from certain countries publish a higher proportion of positive trials than others (58). Excluding non-English studies seemed to modify the results of some reviews (59,lx) but not others (61,62).
Some review teams utilise broad criteria for their inclusion of chief studies (east.chiliad., effects of agents that block the renin-angiotensin organisation on renal outcomes [63]), whereas other teams use more narrow inclusion criteria (e.1000., restricting the analysis merely to patients who have diabetes without evidence of nephropathy [64]). There is often no single right arroyo; however, the conclusions of any meta-analysis that is highly sensitive to altering the entry criteria of included studies should be interpreted with some caution (25). For instance, two different review teams considered whether synthetic dialysis membranes resulted in amend clinical outcomes compared with cellulose-based membranes in patients with astute renal failure. In one meta-analysis (65) merely not the other (66), constructed membranes reduced the take chances for death. The discordant results were due to the inclusion of a study that did non run into eligibility for the 2nd review (67).
Was the Method of Identifying All Relevant Information Comprehensive?
Identifying relevant studies for a given clinical question among the many potential sources of information is commonly a laborious procedure (68). Biomedical journals are the most common source of information, and bibliographic databases are often used to search for relevant articles. MEDLINE currently indexes approximately 4800 medical journals and contains 13 million citations (69). Similarly, EMBASE indexes approximately 5000 medical journals and contains more than than 11 million records. There are some key differences between EMBASE and MEDLINE, and the review squad should take searched both databases (70–72). For example, EMBASE provides the all-time coverage of European research as well as pharmaceutical research including renal adverse events (73). Positive studies may exist more ofttimes published in journals that are indexed in MEDLINE, compared with nonindexed journals (25).
Depending on the question posed, other databases may also have been searched. For case, if a team is summarizing the effects of exercise grooming in patients who receive maintenance hemodialysis, and then searching the Cumulative Index to Nursing and Allied Health Literature (CINAHL) database would be appropriate (74). Alternatively, the ECONOLIT database may exist useful for identifying information on the out-of-pocket expenses incurred past living kidney donors (75). As a supplementary method of identifying information, searching databases such equally the Science Citation Index (which identifies all articles that cite a relevant article), also equally newer Internet search engines such as Google Scholar and Elsevier's Scirus, can be useful for identifying articles that are not indexed well in traditional bibliographic databases (76). Searching bibliographies of retrieved articles tin besides identify relevant articles that were missed.
Whatever bibliographic database was used, the review team should take used a search strategy that maximized the identification of relevant articles (77,78). Because at that place is some subjectivity in screening databases, citations should be reviewed independently and in duplicate by two members of the reviewing team, with the full-text article retrieved for any commendation accounted relevant by any of the reviewers. There is as well some subjectivity in assessing the eligibility of each full-text article, and the risk for incorrectly discarding relevant reports is reduced when two reviewers independently perform each assessment in a reliable way (79).
Important sources of information other than periodical manufactures should not be disregarded. Conference proceedings, abstracts, books, and manufacturers all can be sources of potentially valuable information. Inquiries to experts, including those listed in trial registries, may have also proved useful (28).
A comprehensive search of available literature reduces the possibility of publication bias, which occurs when studies with statistically significant results are more than likely to exist published and cited (80,81). It is interesting that some recent reviews of acetylcysteine for the prevention of contrast nephropathy analyzed as few as v studies, despite being submitted for publication almost one yr after publication of a review of 12 studies (82). Although in that location are many potential reasons for this, one cannot exclude the possibility that some search strategies missed eligible trials. In improver to a comprehensive search method, which makes it unlikely that relevant studies were missed, it is often reassuring if the review squad used graphic and statistical methods to confirm that there was little chance that publication bias influenced the results (83).
Was the Information Abstraction from Each Written report Appropriate?
In compiling relevant information, the review team should accept used a rigorous and reproducible method of abstracting all relevant information from the primary studies. Often ii reviewers abstract key information from each primary study, including study and patient characteristics, setting, and details near the intervention, exposure, or diagnostic test as is appropriate. Language translators may be needed. Teams who conduct their review with due rigor will indicate that they contacted the primary authors from each of the primary studies to ostend the accuracy of bathetic information every bit well as to provide additional relevant information that was non provided in the primary written report. Some authors will go through the additional effort of blinding or masking the results from other study characteristics so that data abstraction is equally objective as possible (84,85).
One element that should take been abstracted is the methodologic quality of each primary study (recognizing this is not always as straightforward equally it may first seem) (86–91). The question to be posed past the reader is whether the reviewing squad considered if each of the chief studies was designed, conducted, and analyzed in a way to minimize or avert biases in the results (92). For randomized, controlled trials, lack of darkening of allocation, inadequate generation of the allocation sequence, and lack of double blinding tin can exaggerate estimates of the handling effect (54,90,93). The value of abstracting such data is that it may assistance to explain important differences in the results among the master studies (90).
For instance, long-term risk estimates can become unreliable when participants are lost to study follow-upwardly; those who participate in follow-up often systematically differ from nonparticipants. For this reason, prognosis studies are vulnerable to bias, unless the loss to follow-up is less than 20% (94). In a systematic review of 49 studies on the renal prognosis of diarrhea associated hemolytic uremic syndrome, on average, 21% of patients were lost to follow-upward (range 0 to 59% across studies) (95). It was hypothesized that patients who were lost to follow-up would contribute to worse estimates of long-term prognosis considering they are typically healthier than those who continue to be followed by their nephrologists. Indeed, studies with a higher proportion of patients lost to follow-upward demonstrated a higher proportion of patients with long-term renal sequelae, explaining 28% of the betwixt-study variability.
How Was the Information Synthesized and Summarized?
In cases in which the master studies differ in the design, populations studied, interventions and comparisons used, or outcomes measured, it may have been appropriate for the review squad simply to report the results descriptively using text and tables. When the primary studies are similar in these characteristics and the studies provide a like guess of a truthful outcome, then meta-analysis may take been used to derive a more than precise estimate of this effect (96). In meta-assay, data from the individual studies are not just combined as though they were from a single written report; rather, greater weights are given to the results from studies that provide more information, because they are likely to exist closer to true effect being estimated. Mathematically combining the results from the individual studies can exist accomplished under the supposition of "fixed" furnishings or "random" furnishings model. Although a thorough description and merits of each approach is described elsewhere (97), it is fair to say that a random-effects model is more than conservative than the fixed-effects arroyo, and a finding that is statistically significant with the latter only non the sometime should be viewed with skepticism.
Whenever private studies are pooled in meta-analysis, information technology is of import for the reader to determine whether it was reasonable to practise then. Ane mode to appraise the similarity of various studies is to inspect the graphic display of the results, looking for similarities in the direction of the estimated issue. Even without considering whatsoever combined meta-analytic issue, a reader becomes much more than confident when a similar consequence is being observed across many studies (i.e., the results have replicated across many studies). Some review teams may report a statistical test to decide how different the studies are from one another (as described previously, this is frequently termed heterogeneity of the study results [98]). This can help to evidence or disprove that differences in the results that were observed between the primary studies is no different from what would be expected by gamble. The most common statistical examination to quantify heterogeneity is something called the Q statistic, which is similar in concept to a χ2 test. Although a nonsignificant effect (by convention P > 0.1) is often taken to betoken that there are no substantial differences between the studies, it is important to consider that this exam is underpowered, especially when the number of studies being pooled is modest. A new statistic that is oftentimes beingness reported in meta-assay these days is something called the Iii statistic. This statistic describes the percent variability between the studies that is present beyond what would exist expected by run a risk. When interpreting an Itwo statistic, values of 0 to 30, 31 to l, and >50% represent mild, moderate, and marked differences between the studies, respectively (99).
Whenever a review squad identifies meaning differences between the primary studies, they should try to explain possible reasons for these differences. This can be done in an informal way past analyzing certain types of studies separately or past selectively combining studies to decide which are especially different from the remaining studies. Alternatively, a statistical approach can be taken to explore differences across studies, using a technique similar to linear or logistic regression (which at the study level is something chosen meta-regression) (100). Either way, a careful exploration of why study results differ can yield important information virtually potential determinants of the effect being observed.
Conclusions
Like all types of research, systematic reviews and meta-analyses accept both potential strengths and weaknesses. With the growth of renal clinical studies, an increasing number of these types of summary publications will certainly go available to nephrologists, researchers, administrators, and policy makers who seek to keep abreast of recent developments. To maximize their advantages, information technology is essential that future reviews be conducted and reported properly, with judicious interpretation past the discriminating reader.
Acknowledgments
A.X.1000. was supported past a Clinician Scientist Award from the Canadian Institutes of Health Research (CIHR). D.H. was supported by a CIHR Fellowship Award, the Chisholm Memorial Fellowship, and the Clinician-Scientist Training Program of the University of Toronto. M.T. was supported by a Population Wellness Investigator Accolade from the Alberta Heritage Foundation for Medical Enquiry and a New Investigator Award from the CIHR.
We thank Drs. Chi Hsu and Harvey Feldman for assistance and advice. We give thanks Arthur Iansavichus, MLIS, who helped compile systematic reviews published in the subject area of nephrology.
Footnotes
-
Published online ahead of print. Publication date bachelor at world wide web.cjasn.org.
- Copyright © 2008 past the American Society of Nephrology
References
- ↵
- ↵
- ↵
- ↵
- ↵
-
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
-
-
- ↵
-
- ↵
- ↵
- ↵
- ↵
-
- ↵
- ↵
- ↵
- ↵
- ↵
-
-
-
- ↵
- ↵
- ↵
- ↵
- ↵
-
-
-
- ↵
-
-
-
-
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
-
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
-
-
-
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
Source: https://cjasn.asnjournals.org/content/3/1/253
Post a Comment for "Potential Strengths and Limitations of a Systematic Review"