This research guide is intended to present useful resources (websites, online tutorials, training videos, key writings) on various aspects of conducting a systematic review, a type of knowledge synthesis among others (meta-analyses, scoping reviews, overviews of reviews, rapid reviews, diagnostic reviews, prognostic reviews, and economic reviews) associated primarily with evidence-based health/biomedicine fields, though increasingly employed in other disciplines. For a related guide, see Scoping Reviews: (Some of) The Basics on this blog.
Definitions of systematic reviews
"Systematic reviews differ from traditional narrative reviews in several ways. Narrative reviews tend to be mainly descriptive, do not involve a systematic search of the literature, and thereby often focus on a subset of studies in an area chosen based on availability or author selection. Thus narrative reviews while informative, can often include an element of selection bias. They can also be confusing at times, particularly if similar studies have diverging results and conclusions. Systematic reviews, as the name implies, typically involve a detailed and comprehensive plan and search strategy derived a priori, with the goal of reducing bias by identifying, appraising, and synthesizing all relevant studies on a particular topic. Often, systematic reviews include a meta-analysis component which involves using statistical techniques to synthesize the data from several studies into a single quantitative estimate or summary effect size (Petticrew & Roberts, 2006). In contrast to traditional hypothesis testing which can give us information about statistical significance (i.e., did the intervention group differ from the control group) but not necessarily clinical significance (i.e., was this difference clinically meaningful or large), effect sizes measure the strength of the relationship between two variables, thereby providing information about the magnitude of the intervention effect (i.e., small, medium, or large). The type of effect size calculated generally depends on the type of outcome and intervention being examined as well as the data available from the published trials; however, some common examples include odds ratios (OR), weighted/standardized mean differences (WMD, SMD), and relative risk or risk ratios (RR). Although systematic reviews are published in academic forums, there are also organizations and databases specifically developed to promote and disseminate them. For example, the Cochrane Collaboration (www.cochrane.org) is a widely recognized and respected international and not-for-profit organization that promotes, supports, and disseminates systematic reviews and meta-analyses on the efficacy of interventions in the health care field" (Uman, 2011).
“A systematic review attempts to collate all empirical evidence that fits the pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimising bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993). The key characteristics of a systematic review are: a clearly stated set of objectives with pre-defined eligibility criteria for the studies; an explicit, reproducible methodology; a systematic search that attempts to identify all the studies that would meet the eligibility criteria; an assessment of the validity of the findings of the included studies, for example through the assessment of the risk of bias; and a systematic presentation, and synthesis, of the characteristics and findings of the included studies” (Green et al., 2008).
"Systematic literature reviews are a method of making sense of large bodies of information, and a means of contributing to the answers to questions about what works and what does not – and many other types of question too. They are a method of mapping out areas of uncertainty, and identifying where little or no relevant research has been done, but where new studies are needed. Systematic reviews also flag up areas where spurious certainty abounds. These are areas where we think we know more than we do, but where in reality there is little convincing evidence to support our beliefs" (Pettigrew and Roberts, 2008).
The gold standard for systematic reviews: The Cochrane Library
Foundational research process for biomedical reviews of all sorts: PICO (What is the PICO Model?) (University of Illinois - Chicago Library)
Components and protocols for conducting systematic reviews: PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)
Registry of systematics reviews in progress: PROSPERO (international database of prospectively registered systematic reviews in health and social care)
Software product for improving healthcare evidence synthesis and the standard production platform for Cochrane Reviews:: Covidence (available through Columbia University Libraries/Health Sciences Library
"A systematic review is a high-level overview of primary research on a particular research question that systematically identifies, selects, evaluates, and synthesizes all high quality research evidence relevant to that question in order to answer it. In other words, it provides an exhaustive summary of scholarly literature related to a particular research topic or question. A systematic review is often written by a panel of experts after reviewing all the information from both published and unpublished studies. The comprehensive nature of a systematic review distinguishes it from traditional literature reviews which typically examine a much smaller set of research evidence and present it from a single author’s perspective. Systematic reviews originated in the biomedical field and currently form the basis of decision-making in Evidence-Based Treatment (EBT) and evidence-based behavioral practice (EBBP)."
- See the NCU Library's Systematic Reviews & Meta-Analysis Workshop Outline and its video workshop (see below)
Five Things to Consider Before Doing A Systematic Review (Centre for Evidence-Based Medicine (CEBM), University of Oxford, 2017)
Useful training videos
Northcentral University Library's Systematic Reviews & Meta-Analyses Workshop.
Databases Typically Searched
- PubMed (CU)
- CINAHL (Cumulative Index to Nursing and Allied Health Literature) Plus with Full Text
- Cochrane Central Register of Controlled Trials (CU)
- Embase (CU)
- PsycINFO (CU)
- Web of Science (CU)
- Scopus (CU)
Key writings on systematic reviews
Antman, E. M., Lau, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992). A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatments for myocardial infarction. Jama, 268(2), 240-248.
Green, S., Higgins, J.P., Alderson, P., Clarke, M., Mulrow, C.D. and Oxman, A.D. (2008). Introduction. In Cochrane Handbook for Systematic Reviews of Interventions (eds J.P. Higgins and S. Green). https://doi.org/10.1002/9780470712184.ch1
Munn, Z., Peters, M.D.J., Stern, C. et al. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 18, 143 (2018). https://doi.org/10.1186/s12874-018-0611-x
Oxman, A. D., & Guyatt, G. H. (1993). The science of reviewing research. Annals of the New York Academy of Sciences, 703(1), 125-134. https://doi.org/10.1111/j.1749-6632.1993.tb26342.x
Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. John Wiley & Sons.
Uman L. S. (2011). Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de l'Academie canadienne de psychiatrie de l'enfant et de l'adolescent, 20(1), 57–59.
Image: Word cloud from Five Things to Consider Before Doing A Systematic Review (Centre for Evidence-Based Medicine (CEBM), University of Oxford, 2017)