(SWiRL-SR ™) Standards

SWiRL-SR™ Systematic Review Librarian

(Standards for Working with Research Librarians on Systematic Reviews ™) Keep “librarians-in-the-loop” ™(Roth, S), to unlock your most powerful tool for high quality systematic reviews, your systematic review librarian! These standards and website are maintained by an experienced medical and systematic review librarian, teacher, educator, writer, and researcher. Stephanie Roth, MLIS (AHIP-D), with over 20 years in the industry. Contact: systematic review librarian @ outlook.com

Sign up to download the SWiRL-SR ™ checklist today and find out how you can keep “librarians-in-the-loop” ™ (Roth, S).

profile
POSTSLINKSPRODUCTSDATABASES/REVIEW TYPES/TOOLS/ABOUT/SWIRL-SRTESTIMONIALS

Roth's Systematic Review Guidebook:

April 2025-last update

Resources & Databases/Review Types

and other sources for systematic reviews:

Additional:

  • ​​PsycNet​ Search and access over 5.4 million records across all of APA’s electronic resources.
  • ​​LILACS​ Scientific and technical literature by Latin America and Caribbean authors.
  • ​​African Index Medicus​ Access to African Health Information
  • Public Health Database This link opens in a new windowProQuest Public Health offers core literature for the study of public health, including biostatistics, environmental health sciences, epidemiology, and behavioral sciences.
  • ​​ERIC​​
  • Google Scholar (Not a database but can still be searched for ​bibliodiversity​)
  • ​​TRIP Database​ (Turning Research Into Practice) is a free medical database providing access to high quality clinical research evidence. TRIP Pro is the enhanced version containing over 100,000 systematic reviews, medical images and videos, millions of full-text articles, and advanced searching.
  • Global Health Archive
    Global Health Archive provides information on international health, biomedical life sciences, non-communicable diseases, public health nutrition, food safety and hygiene, and much more.
  • Transplant Library -Transplant Library is developed by the “Peter Morris Centre for Evidence in Transplantation” and provides access to all evidence-based publications on all aspects of organ transplantation.
  • ​​Epistemonikos​ Provides rapid access to systematic reviews in health and primary studies that have been included in a systematic review.
  • ​​NeuroBite​ (Formerly PsycBite) Catalogues studies of cognitive, behavioral and other treatments for psychological problems and issues occurring as a consequence of acquired brain impairment (ABI).
  • ​​OTseeker​ Occupational Therapy systematic evaluation of evidence.
  • ​​SpeechBite​ Speech pathology database for best interventions and treatment efficacy.
  • ​​CARDS-Computer Access to Research on Dietary Supplements​ Database of federally funded research projects pertaining to dietary supplements since 1999.
  • ​​PDQ-Evidence​ PDQ(“pretty darn quick”)-Evidence facilitates rapid access to the best available evidence for decisions about health systems.

Grey Literature

Systematic Reviews

Guidelines

  • ​​ECRI Guidelines Trust​ Registration is free to access.
  • ​​National Institute for Health and Care Excellence​ The National Institute for Health and Care Excellence (NICE) provides national guidance and advice to improve health and social care.
  • ​​Guidelines-summaries​ Guidelines provides concise clinical guideline summaries of major primary and shared care guidelines in the UK. Additionally, European guidelines from some of the major independent professional bodies are also summarized and included.
  • ​​Canadian Medical Association (CMA) Infobase​ The CMA Infobase is a database of evidence-based Canadian clinical practice guidelines (CPGs).
  • ​​Scottish Intercollegiate Guidelines Network (SIGN)​ The Scottish Intercollegiate Guidelines Network (SIGN) develops evidence based clinical practice guidelines for the National Health Service (NHS) in Scotland.
  • ​​Guidelines International Network (G-I-N)​ The Guidelines International Network (G-I-N) has the world’s largest international guideline library. It is a Network of individual and organisations interested in guidelines.
  • AORN eGuidelines+ AORN eGuidelines+ is a descriptive and comprehensive guidelines collection that reflects evidence-based practices for the perioperative RN in a range of topics that support both patient and worker safety.

​

Critical Appraisal Checklists

Critical Appraisal Checklists

“Critical appraisal skills enable you to assess the trustworthiness, relevance and results of published papers so that you can decide if they are believable and useful.”

(​http://www.casp-uk.net/criticalappraisal​)

Helpful article: ​From critical appraisal to risk of bias assessment: clarifying the terminology for study evaluation in JBI systematic reviews​​

General Checklists

Includes many study designs:

Learn more about Critical Appraisal

Critical Appraisal Checklists by specific Study Design type

Randomized Controlled Trials

​​RoB 2.0 Risk Of Bias Tool​​
CASP: Randomised Controlled Trial Appraisal Tool ​http://media.wix.com/ugd/dded87_40b9ff0bf53840478331915a8ed8b2fb.pdf​​

Non-randomized Studies or Observational Studies:

​​ROBINS-I​ Risk of Bias for non-randomized (observational) studies or cohorts of Interventions

ROBINS I V2 version 2 ​Risk of bias tools – ROBINS-I V2 tool​ for non-randomized studies of interventions

​​ROBINS-E​ Risk of Bias for non-randomized (observational) studies or cohorts of Exposures other than interventions, including environmental and occupational exposures

Newcastle-Ottawa scale (NOS) -for case control or cohort studies

​​http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp​​

IHE Case Series Studies Critical Appraisal Checklist

​​https://www.ihe.ca/research-programs/rmd/cssqac/cssqac-about​​

JBI Critical Appraisal Checklist for Case Series

JBI Critical Appraisal Checklist for Case Reports

The Methodological Index for Non-Randomized Studies (MINOR) is a tool for assessing non-randomized interventional studies-This link will open a PDF document. ​http://cobe.paginas.ufsc.br/files/2014/10/MINORS.pdf​​

Agency for Healthcare Research and Quality (AHRQ) methodological checklist for cross-sectional/prevalence studies ​https://www.ncbi.nlm.nih.gov/books/NBK35156/#_A157879_​​

AXIS-To assess the quality of cross-sectional/prevalence studies-This link will open a PDF document.​ http://bmjopen.bmj.com/content/bmjopen/6/12/e011458.full.pdf​​

The TREND Statement-TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) ​http://www.cdc.gov/trendstatement/​​

The STROBE Statement-STROBE (Strengthening the reporting of observational studies) cohort, case-control, and cross-sectional, conference abstracts ​http://www.strobe-statement.org/index.php?id=available-checklists​​

Mixed Methods Appraisal Tool (MMAT)-used for qualitative, quantitative, and/or mixed methods studies. Helpful for those working on a Mixed Methods Review. This link will open a PDF document. ​ http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf​​

Public Health:

MetaQat – Meta-Tool for appraising all types of public health evidence

​​https://www.publichealthontario.ca/en/Health-Topics/Public-Health-Practice/Library-Services/MetaQAT​​

Qualitative:

CanChild for qualitative reviews

These links will open a PDF document. ​Guidelines​ and ​Form​​

Quantitative Research (Public Health Research)

EPHPP- ​Quality Assessment tool for Quantitative Studies ​​

Animal Research:

Systematic Review Centre for Laboratory Animal Experimentation (SYRCLE) risk of bias assessment tool for assessing animal studies

Article: ​http://www.ncbi.nlm.nih.gov/pubmed/24667063​​

Checklist: ​https://www.radboudumc.nl/getmedia/60b8b9d8-84d7-4325-843c-6408cf065aca/SYRCLE-EFSA-course-1-okt-2016-(quality-assessment)_RoB-table.aspx​​

(invivo studies) Review the ARRIVE guidelines ​https://www.nc3rs.org.uk/arrive-guidelines​​

Critical Appraisal of Studies using Animal Models
​​http://ilarjournal.oxfordjournals.org/content/55/3/405.full.pdf+html​​

Diagnostic Accuracy

QUADAS-2

To evaluate the risk of bias and applicability of primary diagnostic accuracy studies. This link will open a PDF document. ​http://www.bristol.ac.uk/media-library/sites/quadas/migrated/documents/quadas2.pdf​​

​
CASP: Diagnostic Test Studies -This link will open a PDF document. ​http://media.wix.com/ugd/dded87_3815f02af1b34c21b8c3b2b5020024c3.pdf​​

Centre for Evidence Based Medicine (CEBM) Diagnosis Critical Appraisal Worksheet -This link will open a PDF document. ​http://media.wix.com/ugd/dded87_3815f02af1b34c21b8c3b2b5020024c3.pdf​​

​​SIGN checklist for diagnostic accuracy studies​ -This link will open a PDF document.

Library and Information Science Research ​https://www.emerald.com/insight/content/doi/10.1108/07378830610692154/full/html​​

Prognostic Accuracy Studies

QUAPAS ​https://www.latitudes-network.org/tool/quapas/​​

Prediction Models

PROBAST ​https://www.probast.org/​​

Quality Improvement

LibGuide St. Michael’s Unity Health Toronto: Quality Improvement & Patient Safety ​https://guides.hsict.library.utoronto.ca/QI/CA​​

Reliability Studies

​​COSMIN PROMs​​

Grey Literature

​​Public Health Ontario guide to appraising grey literature​​

​

Systematic Review Tools for Systematic Reviews and Comprehensive Reviews

​

Systematic Review Tools

This guide focuses on free or open source tools.

General Tools

Helpful for multiple stages of the review process.

Specific Tools for Planning the Review

Reporting Guidelines: Not sure what to report in your review, locate a reporting guideline.

Protocol Registration: A first step to organize the review and prevent bias is creating and registering a protocol.

Organizing & Collaborating with a Team: Teamwork is core to the systematic review.

Tools for Literature Searching

Citation Manager & Deduplication

Tools for Screening

Did you know screening includes 2 or more reviewers and must be done independent and blinded from the other reviewer(s)?

  • ​​Rayyan​​
  • ​​Rayyan-a Guide for Researchers​​
  • ​​Abstrackr​ Semi-automated screening using machine learning.
  • ​​Sysrev​ Helps facilitate screening, collaborative extraction of data from academic articles and abstracts, PDF documents, and uses machine learning. The free platform supports open access/public projects only. Fee for privacy settings.
  • ​​RobotAnalyst​ RobotAnalyst is designed for searching and screening reference collections obtained from literature database queries. It combines search engine functionality with machine learning and text mining.
  • ​​ASReview​ Utilizes active learning
  • ​​Colandr​ Machine learning, natural language processing, and text-mining functions to partially automate screening.
  • ​​DistillerSR (free with student program)​ Includes AI feature to help rank screening. Predict screening time with time averages.
  • ​​SyRF​ Screening tool for in-vivo or preclinical studies from CAMARADES for systematic reviews of animal data from experimental studies

Data Extraction

​

Critical Appraisal of Clinical Practice Guidelines

​​AGREE II​​

International tool for the assessment of practice guidelines

Critical Appraisal of Systematic Reviews

ROBIS- A tool for assessing the risk of bias in systematic reviews. ​http://www.bristol.ac.uk/social-community-medicine/projects/robis/robis-tool/​​

AMSTAR 2-A Measurement Tool to Assess (quality assessment) of Systematic Reviews

​​http://amstar.ca/​​

​​ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis | BMC Medicine | Full Text​​

RoB NMA for Network Meta-Analysis

​​Risk of Bias in Network Meta-Analysis (RoB NMA) tool | The BMJ​​

​​ICEMAN​ Instrument: Instrument for assessing the Credibility of Effect Modification Analyses ​Development of the Instrument to assess the Credibility of Effect Modification Analyses (ICEMAN) in randomized controlled trials and meta-analyses | CMAJ​​

​​RoB-ME​ A tool for assessing Risk Of Bias due to Missing Evidence in a synthesis

JBI Critical Appraisal for Systematic Reviews Checklist ​(currently not linked since the link has changed frequently)

Health-Evidence Appraisal Tool for Reviews-This link will open a PDF document. ​http://healthevidence.org/documents/our-appraisal-tools/QATool&Dictionary_01Jun16.pdf​​

AHRQ-Assessing the Risk of Bias in Systematic Reviews of Healthcare Interventions

​​https://effectivehealthcare.ahrq.gov/topics/methods-bias-update/methods​​

​​Critical Appraisal of a Mixed Methods Systematic Review​This link will open a PDF document.

​

Guide to Review Types:​

​

What is a Systematic Review?

​Level of Evidence pyramid
​

A systematic review is defined as “a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.” The methods used must be reproducible and transparent.

Source: Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.

Image: EBM Pyramid and EBM Page Generator, copyright 2006 Trustees of Dartmouth College and Yale University. All Rights Reserved. Produced by Jan Glover, David Izzo, Karen Odato and Lei Wang.

When is a Systematic Review the most appropriate study design?

When answering questions of effectiveness comparing two different treatments or interventions.

Choosing a Review Type:

A helpful article about review types. (​Meeting the Review Family: Exploring review types and associated information retrieval requirements​, 2019,Sutton et al.)

7 Stages of Conducting a Systematic Review

1. Gathering your team (Minimum of two reviewers with a third to serve as a tiebreaker)

A systematic review must have a team of two or greater. A systematic review cannot be completed by one person. Choose team members wisely and based on areas of expertise. A third team member is sometimes called a tiebreaker. They are to resolve disagreements for reviewers 1 and 2 for stages of the review that are blinded (i.e. screening, data extraction, critical appraisal) and are completed by two independent reviewers.

2. Questioning (Define a narrow question, may use PICO)

The PICO format is commonly used to define the research question into one that is a searchable question. In some cases, the PICO format may not work and ​another format​ can be used. The P in PICO is Patient/Problem or Person. I is for Intervention/Exposure/Therapy or Treatment. C is optional and is for Comparison (i.e. a placebo, another drug/therapy, gold standard) and O is for Outcome(s), what is the expected or anticipated outcome you will find in the literature? A systematic review question should also be narrow in scope in order to be able to conduct the review in one year. The purpose of a systematic review is to draw conclusions based on the evidence to answer that one well-defined and narrow question.

3. Planning (Create a protocol, plan methods & strategies, register protocol) * The ​Systematic Review Essentials course​ focuses mostly on the planning stage for librarians.

Having a plan in place is essential to a good quality review and by spending more time planning before the review takes place, you could avoid issues or errors that may slow down the process or be detrimental to the review. Planning includes seeing if the review is feasible, checking to make sure there are no conflicting reviews and also ensuring that there is a plan to carry out each stage of the review. Setting goals and timelines for the review is important as well as mapping out how the review project will be managed. This is also put into a document called a protocol. Protocols should be registered (e.g. PROSPERO, OSF Preregistrations, INPLASY) and follow the PRISMA-P for protocol reporting requirements. The protocol also includes defining what the selection criteria will be for the review in terms of inclusion and exclusion criteria for what studies should be screened by for inclusion in the review. This is usually the stage where the librarian should become involved in the systematic review, unless they are conducting their own systematic review and are involved from the start. Librarians involved in the systematic review should be granted co-authorship on the protocol and the final review. However, co-authorship is an individual choice. Librarians can sometimes withdraw their authorship, decisions should always be based on the ​ICMJE.​​

4. Searching/Screening (Exhaustive, transparent & repeatable searching for evidence/selecting studies)

Includes searching multiple databases, grey literature/clinical trial registries, handsearching and supplemental searches (e.g. citation chaining) of the literature (performed by the subject matter expert). Librarians are involved in creating the comprehensive search, translating the search for databases or grey literature, documenting the search and are often involved in deduplicating references in a citation manager and writing the search methods of the review. However, librarians are usually not involved in handsearching unless they are an expert in the subject matter. The review team member with the most subject matter expertise is the one who is best equipped to handsearch. The search stage may also include contacting other experts in the field to identify publications that have not been published yet. Systematic reviews include both published and unpublished literature to avoid a type of publication bias, called positive outcomes bias since positive outcomes are more likely to be published. Screening is done in two phases. The first phase is screening titles/abstracts (together) and the second phase is screening full texts. Screening is done independently by two reviewers, with a third reviewer sometimes serving as a tiebreaker. The process for resolving conflicts should be recorded. If conflicts were resolved by discussion, that should be reported. Reviewers should not move on to the full text screening phase until they have screened all of the titles and abstracts and each is a clear Yes or No without maybes remaining. Once they are ready to screen full texts, they must acquire and read all of the full texts and screen them based on the studies selection criteria. Only Yes’s are included in the review but all No’s must have a reason listed for exclusion. The new PRISMA 2020 requires reporting of study Near Misses too. Near misses are any studies that did not meet inclusion in the review but were very close to being included. Refer to the PRISMA 2020 ​http://www.prisma-statement.org/PRISMAStatement/​ for more guidance on this stage. There are tools designed specifically to assist with the systematic review screening phase (e.g. Covidence, Abstrackr, Rayyan) and ensure a blinded review.

5. Managing & reporting

All methods must be fully reported, transparent and reproducible. The methods reported must also follow the recommended reporting guidance such as the PRISMA 2020. Reporting guidance can be identified by searching the Equator Network ​https://www.equator-network.org/​. Reporting guidance may be modified for review types similar to the systematic review. Refer to the many PRISMA 2020 extensions ​http://www.prisma-statement.org/Extensions/​ for more information.

6. Data Extraction/Synthesizing the evidence

This stage includes appraising the evidence, interpreting results, performing a qualitative (narrative analysis) and/or a quantitative/meta-analysis. A meta-analysis is optional and is only done if it is feasible. A biostatistician or advanced training in statistics is recommended if doing a meta-analysis. There are many tools designed to assist with this process. Evidence from studies are assessed using critical appraisal or Risk of Bias tools/checklists for each of the included study designs.

See my list of ​Critical appraisal checklists​ for more information.

Another resource for systematic review tools for ​appraisal ​is provided by the National Collaborating Centre for Methods and Tools.

Data from all studies must also be extracted and put into tables/charts such as the Summary of Findings (SOF) table and is reported as a narrative synthesis. Data is collected from all studies if conducting a meta-analysis and its numerical findings are reported.

Synthesis: Provide a narrative synthesis of the included studies individually and when combined (What are the differences and the commonality between studies?) or what can be demonstrated from the research when combining the studies together? A meta-analysis is optional. Create a data abstraction/extraction form for the purposes of collecting data that is similar across all included studies, include a ‘Characteristics of Studies’ table to show this data (see table ​example​). Another example of a table is available at ​https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6483779/pdf/CD011479.pdf​ (see page 48). Summary of Findings tables are provided starting on page 8 of the same document. Data extraction must be done using ​data extraction forms ​and conducted by two independent reviewers. There should be a clear process for how to handle disagreements. If disagreements were resolved by discussion, that should be reported. A third reviewer can be called in to resolve any disagreements. Unresolved disagreements should be fully reported.

7. Drawing Conclusions, Writing & Publishing

After completing these steps, the results of the review must be shared. What is the level of evidence? Is there evidence in support of the question or are more studies needed to draw conclusions? What are your recommendations for future studies? What are the limitations to your systematic review? How do these findings from your review change what is known on the topic or question?

Conclusions/Recommendations:

Discuss what contribution this review makes and how your review answers or addresses the original question. Discuss any gaps found in the research. Make recommendations for future research to address these gaps and the importance of addressing them. Discuss the overall strength of evidence in support of your original question (strong, moderate or weak). Remember your systematic review is not just reporting the evidence but creating something new by what the research studies combined tell us.

For more guidance on the systematic review stages, refer to the ​Cochrane Handbook​ (medicine/health sciences), the ​JBI Manual​ (health sciences/nursing) or the methods guides for ​Campbell Collaboration​ Systematic Reviews (Business, education, social welfare, criminal & justice topics and more).

Where to register your Systematic Review Protocol?

Resources:

​

  • JBI Manual: Systematic Reviews of Effectiveness
  • JBI Manual: Systematic Reviews of Prevalence and Incidence
  • JBI Manual: Systematic Reviews for Etiology and Risk
  • JBI Manual: Systematic Reviews of Measurement Properties
  • JBI manual: Systematic Reviews of Text and Opinion
  • How to review the evidence: Systematic identification and review of the scientific literature
    ​
  • ​​BMJ OPEN Systematic Reviews​ Systematic Review articles published in the BMJ Open (Open-Access) medical journal.

​

  • Lockwood, Geum. Systematic reviews: Guidelines, tools and checklists for authors (2017)

Systematic Review Standards

​

What is a Systematic Review?

Systematic Review vs. Literature (Narrative) Review

Traditional literature review / narrative review:

Characteristics:

  • Describes and appraises earlier work but does not describe the specific methods by which the reviewed studies were identified, selected, and evaluated

Uses:

  • Overviews, discussions, critiques of earlier work and the current gaps in knowledge
  • Often used as rationale for new research
  • To scope the types of interventions available to include in a review

Limitations:

  • The writers assumptions and agenda often unknown
  • Biases that occur in selecting and assessing the literature are unknown
  • Can’t be replicated

Systematic review:

Characteristics:

  • The scope of the review is identified in advance (eg review question and sub‐questions and/or sub‐group analysis to be undertaken)
  • Comprehensive search to find all relevant studies
  • Use of explicit criteria to include / exclude studies
  • Application of established standards to critically appraise study quality
  • Explicit methods of extracting and synthesizing study findings (qualitative or quantitative)
  • Sometimes includes a meta-analysis (quantitative synthesis) if applicable

Uses:

  • Identifies, appraises and synthesizes all available research that is relevant to a particular review question
  • Collates all that is known on a given topic and identifies the basis of that knowledge
  • Comprehensive report using explicit processes so that rationale, assumptions and methods are open to scrutiny by external parties
  • Can be replicated / updated

Limitations:

  • Systematic reviews with narrowly defined review questions give specific answers
  • There are remaining questions that the review can’t answer since it is limited to a specific question
  • Requires methodological knowledge of systematic reviews, a lack of knowledge can unknowingly decrease the quality and rigor of the review.
  • It is a significant time investment, a good review will take a year or longer to finish.

Source: Cochrane. Background to Systematic Reviews

What is a Diagnostic Test Accuracy Review?

​

Other name for a Diagnostic Test Accuracy Review?

DTA Review, Systematic Reviews of Diagnostic Test Accuracy

What is a Diagnostic Test Accuracy Review?

“Systematic reviews of diagnostic test accuracy provide a summary of test performance based on all available evidence, evaluate the quality of published studies, and account for variation in findings between studies.” (JBI Handbook)

Diagnostic Test Accuracy Reviews are best designed for:

Assessing the true evidence from diagnostic test accuracy (DTA) studies

What is a Diagnostic Test Accuracy Review? Outline of Stages

Timeframe: 12+ Months

Question: Participants, index test or tests (to evaluate may be old or new), target condition(s) (what you are trying to diagnose), reference standard(s) or reference test (the existing diagnostic test) for verification of index test results (there may be more than one reference standard)

Most DTA review questions are comparative-to replace an existing test, to be used in addition to another test or to decide whether another test should be undertaken (triage)

Sources and searches: Comprehensive & rigorous, includes a combination of database, grey literature and hand searching to locate published and unpublished literature.

Selection: Based on inclusion/exclusion criteria

Appraisal: Critical and rigorous assessment of bias and appraisal using STARD and/or QUADAS-2 or the JBI critical appraisal checklist for diagnostic studies.

Synthesis: Qualitative with Summary of Findings table (SOF) and Quantitative (Meta-Analysis)- varies slightly from a traditional systematic review

*”The Cochrane DTA review structure has three fixed subheadings under the discussion section to guide the interpretation of the results: ‘Summary of main results’ ‘Qualifying DTA evidence’, and ‘Applicability of findings to the review question’. The authors’ conclusions section is divided into ‘Implications for practice’ and ‘Implications for research’.”

This link will open a PDF document.

​​Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy​​

Limitations of a Diagnostic Test Accuracy Review

The unfamiliarity of DTA methods and accuracy metrics makes it difficult to convey results to a wide audience.

Slight variation from traditional systematic review methods.

Requires methodological knowledge of DTA studies.

Resources:

PRISMA-DTA extension statement: ​https://www.prisma-statement.org/dta​​

QUADAS-2 ​Tool​​

STARD ​Statement​​

​

What is an Integrative Review?

​

Other names for an Integrative Review

Integrative Literature Review, Systematic Integrative Review

What is an Integrative Review?

“A review method that summarizes past empirical or theoretical literature to provide a more comprehensive understanding of a particular phenomenon or healthcare problem (Broome 1993). Integrative reviews, thus, have the potential to build nursing science, informing research, practice, and policy initiatives. Well-done integrative reviews present the state of the science, contribute to theory development, and have direct applicability to practice and policy.”

“The integrative review method is an approach that allows for the inclusion of diverse methodologies (i.e. experimental and non-experimental research).”

An integrative review is best designed for:

  • Nursing
  • A review of experimental and non-experimental research simultaneously
    • to define concepts
    • to review theories
    • to review evidence/point out gaps in the literature
    • to analyze methodological issues

Source: Whittemore et al. (2005)

What is an Integrative Review? Outline of Stages

Timeframe: 12+ months. Depends on many factors such as but not limited to: resources available, the quantity and quality of the literature, and the expertise or experience of reviewers” (Grant et al. 2009)

Question: Formulation of a problem, may be related to practice and/or policy especially in nursing.

Sources and searches: Comprehensive but with a specific focus, integrated methodologies-experimental and non-experimental research (all evidence types). ​Purposeful Sampling​ may be employed. Other recommended approaches to searching the literature include ancestry searching, journal hand searching, networking, and searching research registries.” Search is transparent and reproducible.

Selection: Selected as related to problem identified or question, Inclusion of empirical and theoretical reports and diverse study methodologies.

Appraisal: “How quality is evaluated in an integrative review will vary depending on the sampling frame.” Limited/varying methods of critical appraisal and can be complex. “In a review that encompasses theoretical and empirical sources, two quality criteria instruments could be developed for each type of source and scores could be used as criteria for inclusion/exclusion or as a variable in the data analysis stage.”

Synthesis: Narrative synthesis for qualitative and quantitative studies. Data extracted for study characteristics and concept. Synthesis may be in the form of a table, diagram or model to portray results. “Extracted data are compared item by item so that similar data are categorized and grouped together.”

The method consists of:

  • data reduction
  • data display
  • data comparison
  • conclusion drawing,
  • verification

Source: Whittemore et al. (2005)

Limitations of an Integrative Review

  • May be complex due to the combination of incorporating diverse methodologies.
  • Issues related to combining empirical and theoretical reports.
  • Slightly less comprehensive than a systematic review.

Source: ​Whittemore et al. ​(2005)

Sample Integrative Review 1

dos Santos Damásio Silva, Darlan. ​Depression and suicide risk among Nursing professionals: an integrative review.​ Revista da Escola de Enfermagem da U S P. 2015. 49(6): 1023-1031. ISSN: 0080-6234.

Sample Integrative Review 2

Luckett, Tim. Advance Care Planning for Adults With CKD: A Systematic Integrative Review. American journal of kidney diseases. 2014. 5: 761-770. ISSN: 0272-6386

Additional Sources:

Dhollande, S., Taylor, A., Meyer, S., & Scott, M. (2021). ​Conducting integrative reviews: a guide for novice nursing researchers​. Journal of research in nursing : JRN, 26(5), 427–438.

Oermann MH, Knafl KA. Strategies for completing a successful integrative review. Nurse Author Ed. 2021; 31(3-4): 65-68. ​https://doi.org/10.1111/nae2.30​​

​

What is a Living Systematic Review?

​

Other names for a Living Systematic Review

LSR, Live Cumulative Network Meta-analysis (Related), Living Meta-analysis (Related)

*Note: Other review types can be “living” too.

What is a Living Systematic Review?

“We define an LSR as a systematic review which is continually updated, incorporating relevant new evidence as it becomes available.”

  • Are underpinned by continual, active monitoring of the evidence (i.e. monthly searches)
  • Immediately include any new important evidence (meaning data, studies or information) that is identified
  • Are supported by up-to-date communication about the status of the review, and any new evidence being incorporated

Reasons to conduct a Living Systematic Review:

  • High priority (or emerging) question for policy and practice
  • Important uncertainty in the existing evidence
  • Emerging evidence that is likely to impact on what we currently know

Other reasons:

  • To produce evidence that is both trustworthy and current
  • To encourage citizen science, embrace larger collaborations and encouraging open and shared data
  • To provide a comprehensive, always up-to-date review
  • To reduce workload by using experience from previous versions
  • To avoid “research waste”

Sources: Cochrane Collaboration ​http://community.cochrane.org/review-production/production-resources/living-systematic-reviews#resources​ and Cochrane Learning Webinar (see resources) and ​Stay Alive! ​(Winters et al.)

What is a Living Systematic Review? How it differs from a Traditional Systematic Review

Core methods are not different than a systematic review, but they require the “explicit, transparent and predefined decisions on”:

  • How frequently new evidence is sought and screened (ex. need to screen monthly if searching monthly)
  • When and how new evidence is incorporated into the review

Other differences:

  • Author Team: coordinated and continuous effort
  • Meta-analysis: potential statistical adjustments for frequently updating meta-analysis
  • Publication format: persistent, dynamic, online-only (Ex. repository, data repository)
  • Change: Methods and/or topic may change over time
  • Longevity: Thought as to when the review will no longer be living, may not need an LSR forever

Source: Cochrane Collaboration ​http://community.cochrane.org/review-production/production-resources/living-systematic-reviews#resources​ and Cochrane Learning Webinar (see resources)

Limitations of a Living Systematic Review

  • Very time consuming
  • Continuous work flows
  • Team members must have a long term commitment to the project
  • frequent searching and screening
  • frequent statistical analysis-can lead to inflated false-positive rate
  • May require technological tools to support screening, data extraction and critical analysis or risk of bia assessment (machine learning, citizen science)
  • No clear agreement on methods to manage data synthesis
  • Project funding may run out, may need to find creative ways to fund an ongoing LSR to “keep it alive” (Winters, ​Stay Alive!​)
  • Publishing challenge-Deciding where to publish each update and how often, may require multiple DOIs and indexing in Medline, unless publishing to a project website
  • The lifespan varies. There may be an end-date for when the review is no longer updated.

Source: Cochrane Learning Webinar (see resources)

Examples of a Living Systematic Review

Example of a LSR in a journal-The journal publishes regular updates to LSRs:

Cnossen Maryse C., Scholten Annemieke C., Lingsma Hester F., Synnot Anneliese, Tavender Emma, Gantner Dashiell, Lecky Fiona, Steyerberg Ewout W., and Polinder Suzanne. Journal of Neurotrauma. August 2016, ahead of print.​https://doi.org/10.1089/neu.2015.4121​​

What is a Mapping Review?

​

Other names for a Mapping Review

Mapping Evidence Review, Mapping Study, Mapping Review, Mapping Exercise, Systematic Map

What is a Mapping Review?

Mapping reviews are focused on a visual synthesis of the data and are question based rather than topic based like the scoping review.

Mapping reviews are best designed for:

  • When there is an abundance and a diversity of research.
  • As a first step to a systematic review.
  • To identify gaps in a topic area.

What is a Mapping Review? Outline of Stages

Timeframe: 12+ months, (same amount of time as a systematic review or longer) *Varies beyond the type of review. Depends on many factors such as but not limited to: resources available, the quantity and quality of the literature, and the expertise or experience of reviewers” (Grant et al. 2009)

Question: Questions are of a wider scope than a systematic review.
A protocol is registered.

Sources and searches: Searching aimed to give a broad overview, still aims to be thorough and repeatable. Must include a modified PRISMA flow diagram.

Selection: Based on inclusion/exclusion criteria. May require more time spent screening articles due to the larger volume of studies from covering a wider scope. Also necessary to group studies for the mapping of included studies.

Appraisal: None, only if appropriate, includes a quality assessment of study bias/validity.

Synthesis: (Graphical or Tabular, less narrative) Visual synthesis and classification of the available studies. A high level map visualizing the status of the field related to the research question.

(Sources: Petticrew and Roberts (2006), Peterson et al. (2008), Booth et al. (2016).

Limitations of a Mapping Review

  • Similar to those of a ​​scoping review​​​
  • May take time and require additional expertise or training for creating the visual output.

Examples of a Mapping Review

  • ​​​COVID-19-A Living Evidence Map​​ (project ended 10/31/2023)
  • An Evidence map of the effect of Tai Chi on health outcomes (2016)
  • The Use of Technology in the Clinical Care of Depression: An Evidence Map (2018)
  • Lorenc, et al. Crime, fear of crime, environment, and mental health and wellbeing: Mapping review of theories and causal pathways (2012).

​

What is a Meta-Synthesis?

​

Other names for a Meta-Synthesis

Meta-synthesis, Meta-ethnography, Qualitative Evidence Synthesis, Qualitative Meta-Synthesis, Meta narrative review (related)

What is a Meta-Synthesis?

A meta-synthesis is bringing together qualitative data to form a new interpretation of the research field. It helps to build new theories and is not to be confused with a meta-analysis which tests a hypothesis using quantitative data. It primarily generates theory such as program theory, implementation theory, or an explanatory theory of why the intervention works or not, hypothesis for future testing or comparison with trial outcomes.

Meta-syntheses are best designed for:

  • To re-interpret meaning across many qualitative studies.

(Source: S. Atkins et al (2008))

What is a Meta-Synthesis? Outline of Stages

Timeframe: 12+ months or less. *Varies beyond the type of review. Depends on many factors such as but not limited to: resources available, the quantity and quality of the literature, and the expertise or experience of reviewers” (Grant et al. 2009)

Question: May use PICo (Population, Interest, Context), SPICE, SPIDER or PerSPEcTiF question formats, “A clearly formulated question helps to set boundaries for the scope and depth of a meta-ethnography” (Atkins S.)

Sources and searches:

Refer to the 7 S Framework of Qualitative Searching by Booth (2016) ​https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-016-0249-x​​

Search specifically for qualitative studies, includes grey literature. ​Purposeful Sampling​ may be employed.

Selection: Different screening process–includes repeated reading of articles to connect and record concepts or themes.

Appraisal: Focus is on the translation of studies and then systematically compared or ‘translated’ within and across studies while retaining the structure of the relationships between central concepts/themes and includes a ‘thematic analysis’.

Synthesis: Qualitative Differs from a Meta-Analysis (Quantitative)“The goal is not aggregative in the sense of ‘adding studies together’ as with a meta-analysis. On the contrary, it is interpretative in broadening understanding of a particular phenomenon.” (Source: Grant et al (2009))

There are 3 types of synthesis that may be used. A second level of synthesis is possible.

  1. Reciprocal Translation: Concepts in one study can incorporate those of another.
  2. Refutational Translation: Concepts in different studies contradict one another.
  3. Line of Argument Synthesis: Studies identify different aspects of the topic that can be drawn together in a new interpretation.

(Source: France EF, Ring N et al (1988))

For more about the reporting guidelinesfor a Meta-Synthesis or Meta-Ethnography please visit theeMERGe Project website at ​http://emergeproject.org/​ and ​ENTREQ​for the reporting of qualitative evidence syntheses.

There is also the Cochrane Qualitative & Implementation Methods Group: ​http://methods.cochrane.org/qi/​​

SPICE question format

​​spice​​​Download​​

SPIDER question format

​​spider​​​Download​​

PerSPEcTiF question format

​​perspectif​​​Download​​

​​PICO and Beyond Template​– For writing a question using formats other than PICO

Resources:

Atkins S, Lewin S, Smith H, et al. ​Conducting a meta-ethnography of qualitative literature: lessons learnt​. BMC Medical Research Methodology. 2008; 8:21.

Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Newbury Park, California: Sage Publications, Inc.; 1988.

​​Cochrane Qualitative and Implementation Method Group guidance series​​

  • Cochrane Qualitative and Implementation Methods Group guidance series—paper 1: introduction
  • Cochrane Qualitative and Implementation Methods Group guidance series—paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis
  • Cochrane Qualitative and Implementation Methods Group Guidance Paper 3: Methods for assessing evidence on Intervention Implementation
  • Cochrane Qualitative and Implementation Methods Group guidance series—paper 4: methods for assessing evidence on intervention implementation
  • Cochrane Qualitative and Implementation Methods Group guidance series—paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews
  • Cochrane Qualitative and Implementation Methods Group guidance series—paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses

The seven phases of Noblit and Hare’s meta-ethnography (meta-syntheses) approach

​The seven phases of Noblit and Hare's meta-ethnography approach
​

(Source: France EF, Ring N et al (1988))

Limitations of a Meta-Synthesis

​

  • Requires a different skill set than a traditional systematic review of quantitative studies
  • Choice of a meta-ethnography may not be confirmed until pool of evidence known
  • Requires significant methodological skill and experience with qualitative methods
  • May take time to engage with the evidence and develop theory
  • Requires further interpretation by policy makers and practitioners

(Source: M. Petticrew et al (2013) and Li T. et al (2001))

Oops, if you find any broken links, please ​contact me​!

​

What is a Mixed Methods Review?

​

Other names for a Mixed Methods Review

Mixed Research Synthesis, Mixed Methods Synthesis, Mixed Methods Systematic Review, Mixed Papers Review, Mixed Studies Review, Qualitative & Quantitative Systematic Review

What is a Mixed Methods Review?

Mixed Methods Reviews “broaden the conceptualization of evidence, [are] more methodologically inclusive and produce syntheses of evidence that will be accessible to and usable by a wider range of consumers.�? (Sandelowski et al. (2012))

“Mixed-methods systematic reviews can be defined as combining the findings of qualitative and quantitative studies within a single systematic review to address the same overlapping or complementary review questions.” (Harden A. 2010)

“The mixed methods approach to conducting systematic reviews is a process whereby (1) comprehensive syntheses of two or more types of data (e.g. quantitative and qualitative) are conducted and then aggregated into a final, combined synthesis, or (2) qualitative and quantitative data are combined and synthesized in a single primary synthesis.” (The Joanna Briggs Institute 2014 Reviewers Manual)

Mixed Methods Reviews are best designed for:

  • Multidisciplinary topics or topics with a body of literature that includes quantitative, qualitative, and mixed methods studies.
  • To determine not only the effects of interventions but also their appropriateness.
  • To identify research gaps.
  • To provide an explanation for possible heterogeneity between trials.
  • To answer multiple questions in one systematic review.
  • “Integrate quantitative estimates of benefit and harm with more qualitative understanding from people’s lives.” (Harden A. 2010)

What is a Mixed Methods Review? Outline of Stages

Timeframe: 12-18+ months. Same as a systematic review or longer. Depends on many factors such as but not limited to: resources available, the quantity and quality of the literature, and the expertise or experience of reviewers” (Grant et al. 2009)

Question: Addresses 2 or more specific PICO criteria, “Mixed methods reviews should pose a question that specifically requires the inclusion of two or more syntheses that are grounded in different approaches”. A protocol is registered.

Examples of and clearly articulated PICO questions that may be posed by one mixed methods systematic review are:

1. What is the effectivenessof educational strategies associated with insulin pump therapy?

2. What is the appropriateness of educational strategies associated with insulin pump therapy�?

Sources and searches: Types of studies and their findings are mixed. This requires a very broad search or multiple structured searches. “This should address each of the syntheses included in the review…[and] aims to find both published and unpublished studies.” ​Purposeful Sampling​ may be employed.

Selection: Based upon inclusion criteria.

  • “The quantitative component of the review will consider any experimental study design including randomized controlled trials, non-randomized controlled trials, quasi-experimental, before and after studies for inclusion.”
  • “The qualitative component of the review will consider studies that focus on qualitative data including, but not limited to, designs such as phenomenology, grounded theory, ethnography, action research and feminist research. In the absence of research studies, other text such as opinion papers and reports will be considered. The textual component of the review will consider expert opinion, discussion papers, position papers and other text. • The economic component of the review will consider cost effectiveness, cost benefit, cost minimization, cost utility.”

Appraisal: “This should address each of the syntheses included in the review.” (Example: Quantitative, Qualitative, Textual or Economic papers) The appropriate critical appraisal tool or method should be applied by study type or there should be one general tool used that will represent all study designs.

​​Mixed Methods Appraisal Tool (MMAT)​This link opens a PDF document.

Synthesis: Depends on findings and included studies, applying mixed synthesis methods (qualitative: thematic synthesis/meta-ethnography & quantitative: statistical meta-analysis). “The results of each single method synthesis included in the mixed method review will be extracted in numerical, tabular or textual format.”

(Source: JBI 2014)

Limitations of a Mixed Methods Review

  • May result in a larger number of citations.
  • Requires more search time to create multiple searches for varying outcomes.
  • Requires significant methodological skill in both quantitative and qualitative research.
  • Resource intensive–may take time to engage with the evidence and develop theory.
  • Highly iterative nature of the interpretative process.

(Source: M. Petticrew et al. 2013)

​

What is an Overview of Reviews?

​

Other names for an Overview of Reviews

Systematic Review of Systematic Reviews, Umbrella Review, Review of Reviews, Summary of Systematic Reviews, Synthesis of Reviews, Review of Systematic Reviews, Review of Reviews, Review of Meta-analyses, Meta-review, Systematic Meta-review, Meta-Meta-analysis

What is an Overview of Reviews?

A review of reviews or systematic review of systematic reviews. “The intent of this kind of review is to include [other reviews], systematic reviews or meta-analyses/as the main study type and thus examine only the highest level of evidence.” Examines two or more systematic reviews or evidence syntheses.

Source: Blackwood D (2016)

Overviews of Reviews are best designed for:

  • Synthesizing and combining relevant data from existing systematic reviews or meta-analyses to make better decisions.
  • To provide clinical decision makers with the evidence they need when there are too many systematic reviews for them to keep up with for an intervention.

What is an Overview of Reviews? Outline of Stages

Timeframe: Approximately 12+ months. *Varies beyond the type of review. Depends on many factors such as but not limited to: resources available, the quantity and quality of the literature, and the expertise or experience of reviewers” (Grant et al. 2009)

Question: Overviews are broader in scope but the question is still specific and well-defined (may differ depending on the type of overview), can use PICO to formulate the question (may differ depending on the review types included).

Sources and searches: Only locating the highest level of evidence of published or unpublished:

  1. Systematic Reviews or Meta-analyses
  2. Other types of evidence synthesis that meets the predefined inclusion criteria

Only identifies other reviews or evidence syntheses.

Selection: Based upon clear inclusion/exclusion criteria and outcome measures.

Appraisal: Critical Appraisal of systematic reviews done by at least two independent reviewers using the appropriate tool for the type of review.

Synthesis: Present outcome data exactly as they appear or re-analyze the outcome data in a different way.

Limitations of an Overview of Reviews

  • Should not summarize but should synthesize and integrate the evidence of other systematic reviews.
  • An Overview of Reviews in not easier than a regular systematic review, same intensity and rigor as a traditional systematic review.

Resources:

Cochrane Collaboration: ​Overviews of Reviews ​​

JBI Manual: ​Umbrella Reviews​​

Aromataris E, Fernandez R, Godfrey C, et al. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. International journal of evidence-based healthcare. 2015;13:3.

Smith V, Devane D, Begley C and M Clarke. ​Methodology in conducting a systematic review of systematic reviews of healthcare interventions​. BMC Medical Research Methodology. 2011;11:15.

​​Overviews of systematic reviews: development and evaluation of methods:​ Thematic Series published in BMC Systematic Reviews

What is a Rapid Review?

​

Other names for a Rapid Review

Rapid Evidence Review, Rapid Evidence Assessment, Rapid Systematic Review, Expedited Review, Rapid Evidence Summary

*Note: Other review types can be “rapid” too.

What is a Rapid Review?

“Rapid reviews are a form of evidence synthesis that may provide more timely information for decision making compared with standard systematic reviews.” (AHRQ) The methods of conducting rapid reviews varies widely, and are typically done in less than 5 weeks. Often policy makers require a short deadline and a systematic review for synthesizing the evidence is not practical. A rapid review speeds up the systematic review process by omitting stages of the systematic review making it less rigorous.

Rapid Reviews are best designed for:

New or emerging research topics, updates of previous reviews, critical topics, to assess what is already known about a policy or practice using some systematic review methods.

What is a Rapid Review? Outline of Stages

Timeframe: ≤ 5 weeks (varies) *Varies beyond the type of review. Depends on many factors such as but not limited to: resources available, the quantity and quality of the literature, and the expertise or experience of reviewers” (Grant et al. 2009)

Question: Narrow question, may use PICO

Sources and searches: Sources are limited due to time constraints of searching, however still uses transparent and reproducible search methods.

Selection: Based on inclusion/exclusion criteria

Appraisal: Critical and rigorous but time limited

Synthesis: Descriptive summary or categorization of data, may still be quantitative

(Source: Khangura S. et al. (2012) Evidence summaries: the evolution of a rapid review approach)

Resources:

Advances in Rapid Reviews series (Systematic Reviews BMC) ​http://www.biomedcentral.com/collections/arr​​

Rapid Review Guidebook-(from the National Collaborating Centre for Methods and Tools) This link will open a PDF document. ​http://www.nccmt.ca/uploads/media/media/0001/01/​​

​​STARR Decision Tool​: SelecTing Approaches for Rapid Reviews (STARR) Decision Tool

Ganann R, Ciliska D and Helen T. ​Expediting systematic reviews: methods and implications of rapid reviews.​Implementation Science. 2010; 5:56.

Hartling L, et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. J Clin Epidemiol. 2015; 68(12).

Tricco AC, Antony J, Zarin W, et al. A scoping review of rapid review methods. BMC Medicine. 2015;13:224.

Rapid Review Examples to rapidly synthesize the literature during COVID-19:

  • Digital contact tracing technologies in epidemics: a rapid review (Cochrane)
  • Quarantine alone or in combination with other public health measures to control COVID-19: a rapid review (Cochrane)

Limitations of a Rapid Review

  • Search is not as comprehensive
  • In come cases, there may only be one reviewer.
  • Limited/cautious interpretation of the findings
  • Potential for limitations and/or biases.
  • May still require advanced knowledge of the systematic review stages since many of the same stages are conducted.
  • Results should not be overstated due to the rapid evidence assessment.

(Source: Cochrane: Rapid Reviews-An Introduction (2014))

What is a Scoping Review?

​

Other names for a Scoping Review

Scoping Study, Systematic Scoping Review, Scoping Report, Scope of the Evidence, Rapid Scoping Review, Structured Literature Review, Scoping Project, Scoping Meta Review

What is a Scoping Review?

“Scoping reviews have great utility for synthesizing research evidence and are often used to [categorize or group] existing literature in a given field in terms of its nature, features, and volume.” Often a scoping review is confused with a ​mapping review​. Their are subtle differences, like scoping reviews are more topic-based and mapping reviews are more question-based.

According to Grant and Booth (2009), Scoping reviews are “preliminary assessment of potential size and scope of available research literature. Aims to identify nature and extent of research evidence (usually including ongoing research).”

Scoping Reviews are best designed for:

“When a body of literature has not yet been comprehensively reviewed, or exhibits a large, complex, or heterogeneous nature not amenable to a more precise systematic review.”

  • Label body of literature with relevance to time, location (e.g. country or context), source (e.g. peer-reviewed or grey literature), and origin (e.g. healthcare discipline or academic field)
  • Clarify working definitions and conceptual boundaries of a topic or field
  • Identify gaps in existing literature/research

(Peters M, Godfrey C, Khalil H, et al)

​

What is a Scoping Review? Outline of Stages

Timeframe: 12+ months, (same amount of time as a systematic review or longer) *Varies beyond the type of review. Depends on many factors such as but not limited to: resources available, the quantity and quality of the literature, and the expertise or experience of reviewers” (Grant et al. 2009)

Question: Answers broader and topic focused questions beyond those relating to the effectiveness of treatments or interventions. A protocol is registered.

Sources and searches: Comprehensive search-may be limited by time/scope restraints, still aims to be thorough and repeatable of all literature. May involve a broad search or multiple structured searches. This may produce more results than a traditional systematic review.

Selection: Based on inclusion/exclusion criteria, due to the iterative nature of a scoping review some changes may be necessary. May require more time spent screening articles due to the larger volume of results from broader questions. Must include a modified PRISMA flow diagram and be reported according to the PRISMA-SCR extension.

Appraisal: Critical appraisal (optional), Risk of Bias assessment (optional) may not be applicable for scoping reviews.

Synthesis: (Tabular with some narrative) The extraction of data for a scoping review may include a charting table or form but a formal synthesis of findings from individual studies and the generation of a ‘summary of findings’ (SOF) table is not required. Results may include a logical diagram or table or any descriptive form that aligns with the scope and objectives of the review. May incorporate a numerical summary and qualitative thematic analysis.

(Sources: MDJ Peters et al. (2015), Levac et al. (2010))

Limitations of a Scoping Review

  • Is not easier than a systematic review.
  • Is not faster than a systematic review, may take longer.
  • More citations to screen
  • Different screening criteria/process than a systematic review
  • Often leads to a broader, less defined search or multiple structured searches
  • Increased emphasis for hand searching the literature.
  • May require larger teams because of larger volume of literature.

Resources:

Munn Z, Peters M, Stern C, et al. ​Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach​. BMC Medical Research Methodology. 2018;18:143.

Peters, Micah D.J., Godfrey, Christina; McInerney, Patricia; Khalil, Hanan; Larsen, Palle; Marnie, Casey; Pollock, Danielle; Tricco, Andrea C.; Munn, Zachary ​Best Practice Guidance and Reporting Items for the Development of Scoping Review Protocols​. JBI Evidence Synthesis, February 09, 2022.

Peters M, Marnie C, Tricco A, et al. ​Updated Methodological ​Guidance for the Conduct of Scoping Reviews. JBI Evidence Implementation. 2021;19:3-10.

Peters M, Godfrey C, Khalil H, et al. ​Guidance for Conducting Systematic Scoping Reviews​. Int J Evid Based Healthc. 2015;13:141-146.

Morris M, Boruff J, and G Gore. Scoping Reviews: establishing the role of the librarian. J Med Lib Assoc. 2016;104(4):346-353.

Khalil H, Peters M, Godfrey C, et al. An Evidence-Based Approach to Scoping Reviews. Worldviews on Evidence-Based Nursing. 2016;00:0,1-6.

Arksey H. and O’Malley L. ​Scoping Studies: towards a methodological framework​. International Journal of Social Research Methodology, 2005;8(1):19-32.

Pham MT, Rajic A, Greig JD, et al. A Scoping Review of Scoping Reviews: advancing the approach and enhancing the consistency. Research Synthesis Methods. 2014;5(4):371-385.

Levac D, Colquhoun H, O’Brien K. ​Scoping Studies: Advancing the Methodology.​Implementation Science. 2010; 5(1).

​

What is a Network Meta-Analysis (NMA)?

​

What is a Network Meta-Analysis (NMA)?

Other names for a Network Meta-Analysis

NMA, MTC Meta-Analysis, Multiple Treatment Meta-Analysis, Mixed Treatments Comparison, Multiple Treatments Comparison, Pair-Wise Meta-Analysis, Indirect Treatment Comparison, Multiple Treatment Comparison Meta-Analysis, Live Cumulative Network Analysis (New type)

What is a Network Meta-Analysis (NMA)?

“Network meta-analysis compares multiple interventions simultaneously by analyzing studies making different comparisons in the same analysis.”

Source: M. Petticrew et al. (2013)

Network meta-analyses are best designed for:

  • Conditions with multiple interventions (3 or more)
  • Draw direct and indirect comparisons
  • To answer more relevant clinical questions
  • To make treatment estimates for an entire treatment network instead of scanning each individual pair-wise comparison
  • To give the “full picture” to clinicians
  • Gain precision by considering all available evidence, not just (A vs. B comparisons); accounting for transitivity.
  • Potential to more explicitly “rank” treatments using summary outputs

What is a Network Meta-Analysis? Outline of Stages

Timeframe: 12-18+ months. Same as a traditional systematic review. *Varies beyond the type of review. Depends on many factors such as but not limited to: resources available, the quantity and quality of the literature, and the expertise or experience of reviewers” (Grant et al. 2009)

Question: PICO for multiple interventions or treatments (3 or more). Combines indirect (triangular approach) or direct comparisons (pairwise approach).

Sources and searches: Requires a large search to locate all of the trials using transparent & reproducible methods.

Selection: Same as a systematic review–based upon clear inclusion/exclusion criteria. Requires a lot more screening of trials.

Appraisal: Formal quality assessment of all studies. Still includes Risk of Bias and quality of evidence assessments. Refer to GRADE and CINeMA for measuring the certainty and confidence of the evidence.

Synthesis: Statistical analysis when possible (heterogeneity a potential problem with indirect comparisons)–uses different statistical methods than a systematic review. Creates a web-like analysis called a Network Diagram or Network Comparison. May use bayesian frameworks or meta-regression analysis.

Limitations of a Network Meta-Analysis

  • Requires specialist statistical expertise and software
  • Assumes that all interventions included in the “network” are equally applicable to all populations and contexts of the studies included.
  • May introduce study selection bias.
  • Ease of software may lead researchers to undertake this type of review even though it may not be the most appropriate study design for their research question.
  • Complex methodological approaches

Source: M. Petticrew et al. (2013) and Li T. et al. (2011)

Example of a published Treatment Network for the Drugs Considered in the Example Multiple Treatment Comparison on Generalized Anxiety Disorder

Source:

Mills, Edward J., John P. A. Ioannidis, Kristian Thorlund, Holger J. Schünemann, Milo A. Puhan, and Gordon H. Guyatt. 2012. How to use an article reporting a multiple treatment comparison meta-analysis. Jama 308 (12): 1246-53. ​ http://jamanetwork.com/journals/jama/fullarticle/1362030​​

Resources:

Assessing the Feasibility of a Network Meta-Analysis (​VIDEO​)

Petticrew M, Rehfuess E, Noyes J, et al. Synthesizing evidence on complex interventions: How meta-analytical, qualitative, and mixed method approaches can contribute. J Clin Epid. 2013;66:1230-1243.

Chapter 11-Cochrane Handbook: ​Undertaking network-meta-analyses​​

Li T, Puhan MA, Swaroop SV, Dickersin K, et al. ​Network meta-analysis-highly attractive​ ​but more methodological research is needed.​BMC Medicine. 2011.

PRISMA NMA for ​Network Meta-Analyses ​​

​​RoB NMA​-Risk of Bias in Network Meta-Analysis (use in combination with AMSTAR 2 and ROBIS to critically appraise the systematic review methods)

GRADE -for NMA to measure certainty of the evidence

​​A guide and pragmatic considerations for applying GRADE to network meta-analysis | The BMJ​​

CINeMA (Confidence in Network Meta-Analysis)

​​CINeMA​​

​​https://cinema.ispm.unibe.ch​​

Cochrane training: ​CINeMA – Confidence in network meta-analysis | Cochrane Training​​

ROB-MEN

​​ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis | BMC Medicine | Full Text​​

​​ICEMAN​ Instrument: Instrument for assessing the Credibility of Effect Modification Analyses ​Development of the Instrument to assess the Credibility of Effect Modification Analyses (ICEMAN) in randomized controlled trials and meta-analyses | CMAJ​​

Cochrane Network Meta-analysis ​Toolkit ​​

Link to ​Published​ Network Meta-Analyses from Cochrane Methods (around 2015)

Example of a Cochrane NMA from 2025-

​​https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD012859.pub2/​​

Graphical Tools for Network Meta-Analysis in ​STATA ​:

What is a Live Cumulative (NMA) Network Meta-Analysis:

An NMA that is updated at regular intervals via crowdsourcing of a research community.

​​http://livenetworkmetaanalysis.com​​

​

Open Access Health Resources

​Open Access Health Resources

*Check the individual licenses for exact reuse guidelines of the sources listed on this page.

OA Catalogs

  • ​​OAIster​ This link opens in a new windowOAIster serves as a search portal for digital collections of electronic books, online journals, audio and image files, and movies provided by the research library community. Users can search by keyword, title, creator, subject or resource type. OAIster is a union catalog of digital records built by harvesting records using the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH).

OA Search Engines

  • ​​Science.gov​ Gateway to government science information and research results.
  • ​​Evidence Aid​ Evidence Aid provides free access to published systematic reviews of relevance to natural disasters, humanitarian crises or major healthcare emergencies, that include health outcomes.
  • ​​CORE ​Search across hundreds of open repositories from around the world.

Open Data Sharing

​

Preprints /OA Self-Archived Directories

OA Repositories/Repository Directories

Additional OA Sources

Quick Tips to avoid publishing in a predatory journal or presenting at a predatory conference

  • Trust your instincts
  • Look at the quality of the articles published in the journal or the quality of the conference website
  • Check to see if the journal is ​currently indexed ​in PubMed (NLM) and included in ​DOAJ​ or has the ​DOAJ Seal​​
  • ​​Think. Check. Submit.​ or ​Think. Check. Attend. ​(for conferences)
  • Be alert to spam emails or calls that may be a scam (usually start with Dear esteemed scholar…), just delete them
  • Google the journal or conference to find out what others are saying.
  • Don’t be too trusting! Don’t be afraid to validate affiliations, editorial board, and impact factors.
  • If it’s sounds too good to be true, be distrustful. Peer review takes time, a quick turnaround time may actually hurt your reputation as a scholar. Be careful to vet everything that will have your name attached to it, you may not be able to remove it later.
  • When in doubt contact your local medical librarian to check for you.

​

​Systematic Review Librarian
​

Systematic Review Librarian/SWiRL-SR

Created by Medical Librarian, Stephanie C. Roth, MLIS (AHIP-D)

Contact: systematic review librarian @ outlook .com
​

Stephanie Roth is the copyright owner of this website. Content cannot be resold, shared or reproduced without permission of the copyright owner.

​

​

Last updated: April 2025

Broken Link? Email Systematic Review Librarian @ outlook .com
​