Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Systematic Reviews and Other Expert Reviews

#systematicreviews #sysrev

Initiate Your Review

Prior to starting your project, you should search of the literature to determine if a review has recently been published on your topic of interest.  Depending on the quality of a recent review, you may decide your planned review is not needed at this time and/or that your review question should change.

For help with searching the literature for recent reviews, contact your Welch Medical Library Informationist

A well-thought-out and clearly formulated research question is the foundation for a successful systematic review.  It provides the basis for the search terms used to identify evidence and informs the generation of inclusion/exclusion criteria used to select evidence.  A research question includes multiple components, which may include a population, an exposure or intervention, a setting or context, a comparison, an outcome, and a timeframe.  The inclusion/exclusion criteria specifically address these components and are used to decide which studies will be included in or excluded from the review.

For more information about the process of defining the research question and developing criteria for inclusion see Core Methods section of the Cochrane Handbook.

The PICO (Patient or Problem, Intervention, Comparison, Outcome) framework is a commonly used tool for formulating research questions. Davies (2011) reviews PICO and numerous PICO-derived frameworks for developing evidence based practice questions.

Systematic reviews are specialized research projects that require multiple types of expertise.  These include subject matter knowledge, familiarity with systematic review methodology, skills for searching in a variety of databases, and expertise in statistical methods.

The inclusion of a trained information professional on the team has become part of established standards for high quality systematic reviews ( IOM, Cochrane, AHRQ).  Contact your Welch Informationist about becoming a part of your team.

A protocol is a key document that outlines how the many steps of a systematic review will be carried out.  It provides a roadmap to the review team and shows transparently how the review will be conducted.

It's a good idea to register your protocol.  It increases transparency, minimizes bias, and reduces the redundancy of groups working on the same topics (PLoS Medicine Editors, 2011).

The following registries publish protocols of systematic reviews and include information about registering a protocol:

Contact your Welch Informationist with questions about developing your protocol and where to register it.

The following resources list commonly used frameworks for developing research questions:

The following are free tools for analyzing topics and visualizing connections within large sets of documents:

Identification of Evidence

Databases must be searched as comprehensively as possible in order to identify all of the documents relevant to your systematic review.  The following components are used in comprehensive searching:

  • Controlled vocabulary terms (e.g. as found in PubMed's MeSH Database)
  • Keywords (also called natural language terms or text words)
  • Boolean operators (AND, OR) and/or proximity operators
  • Database-specific syntax that allows for the searching of phrases, parts of documents, variants of terms, etc.

Visit our Expert Searching guide to learn more about these and other searching concepts. 

When Welch Medical Library Informationists are on systematic review teams, they develop comprehensive search strategies for multiple databases.

When searching for health and medical information you are expected to search PubMed, which includes the MEDLINE database of indexed citations. PubMed contains a very large collection of literature, but to be comprehensive you will need to supplement your PubMed search with additional databases. Always check the requirements or recommendations specified in guidelines for the type of review you're conducting (See this guide's "Standards and Guidelines" page for a list of guidelines).  

Things to consider when selecting additional databases include

  • subject area(s),
  • regional coverage, and
  • language coverage.

The Welch Medical Library website offers a searchable and browsable list of Hopkins-subscribed databases. This list covers topics including biomedicine, health, engineering, business, economics, criminal justice, public policy, etc. In addition, the Hopkins Sheridan Libraries maintain a collection of subject guides that link to subject-specific databases.

Search filters, also known as "search hedges," are pre-developed search strategies for topics and study types. We recommend only using validated filters that have been tested for reliability and accuracy.  

Do not confuse filters with pre-set limits in databases, even when these limits are called filters (as they are in PubMed). Pre-set limits such as "Article types," "Humans," and "Ages" will limit the search to controlled vocabulary terms, and exclude articles that would be found with the combination of controlled vocabulary and keywords used by validated filters.

See this guide's "Expert Searching Tips" page for more information about search filters, including examples of validated filters for finding randomized controlled trials. Contact your Welch Informationist for questions about the appropriate use of search filters.

Grey literature is information produced by government agencies, academic institutions, and the for-profit sector that is not made available by commercial publishers. This kind of literature can be more difficult to find than the journal literature, but you may wish to include it because it may contain unique and relevant evidence related to your review.  

Examples of grey literature document types have been compiled by GreyNet International and include clinical trials, reports, proceedings, dissertations and theses, white papers, newsletters, and patents. Some of these types of documents can be found along with the journal literature in some databases, but grey literature searches often require the use of other specialized sources.

Contact your Welch Informationist for questions about grey literature and how to search for it.

Handsearching involves the page by page examination of key journals or conference proceedings for relevant documents.

Handsearching may identify documents that are missed by database searching because of how articles are described, or indexed, in databases such as PubMed or Embase. Gaps in indexing may result from whole journals not being indexed, portions of journals not being indexed, lack of appropriate indexing terms, inconsistencies in indexing, or poorly reported research that does not allow accurate indexing (Hopewell, Clarke, Lefebvre, & Scherer, 2007). 

Citation searching involves manually exploring the citation networks of selected relevant articles as another way of identifying documents that may have been missed through other means of searching. Forward searching identifies later articles that have cited articles of interest.  Backward searching identifies earlier articles from the reference lists of articles of interest. The databases Scopus and Web of Science have excellent features for citation searching.

Contact your Welch Informationist with questions about best practices for handsearching and citation searching.

The following are free tools for identifying MeSH terms and keywords from text or articles:

Selection of Evidence

A citation management tool is needed to organize the results of searches and to remove and store duplicates. Typically, records are then exported from the citation management tool to a screening tool.

A pre-established set of inclusion/exclusion criteria are employed when screening studies.

In systematic reviews, the selection of studies is a two-step process that includes a title/abstract review followed by a full-text review. Studies are reviewed according to the inclusion/exclusion criteria by two reviewers, who must agree to include or exclude. Any conflicts are resolved by a third reviewer.

EndNote is the preferred tool for large systematic reviews because of the following features:

  • Bulk import of large files
  • Advanced options for removing duplicate records
  • Bulk retrieval of full text

RefWorks is sometimes used for smaller reviews for the following reasons:

  • An institution-wide subscription for the Johns Hopkins community
  • Collaborative features

Information about these and other citation management tools are available from the library's Citation Management guide.

Tools that are Best Used for Screening (Basic Data Abstraction Features, If Any):
  • Covidence - Johns Hopkins has an institution-wide subscription and the Welch Medical Library provides support for Covidence.
  • abstrackr (free)
  • Rayyan (free)
Tools that Offer More Advanced Features (Beyond Screening):

Additional tools can be found by searching the Systematic Review Toolbox (SR Toolbox).

Evaluation of Evidence

Data abstraction is the systematic collection of data elements from included studies in a review. Data elements could include study design, intervention, outcomes measured, results, etc.  This is best achieved through a standardized form derived from your research question and inclusion/exclusion criteria.  Pilot the use of this form before full implementation.

Study appraisal systematically examines factors such as

  • the appropriateness of study design,
  • outcome measures,
  • methodological quality and the risk of bias, and
  • the quality of reporting.

Tools that Have Robust Data Abstraction Features:

  • DistillerSR ($)
  • EPPI-Reviewer ($)
  • HAWC - Health Assessment Workspace Collaborative (free) - developed for toxicology
  • Qualtrics - (licensed through Hopkins) survey tool for data capture
  • REDCap - (licensed through Hopkins) Research Electronic Data Capture
  • SRDR - Systematic Review Data Repository (free)
  • Systematic Review Facility (free) - developed for animal studies

Additional tools can be found by searching the Systematic Review Toolbox (SR Toolbox).

Tools for Measuring Study Quality:

Analysis and Interpretation of Evidence

The synthesis of evidence includes a qualitative synthesis of included studies which describes study methodology, strengths and limitations, patterns across studies, potential bias in study design, and the relevance of studies to the populations, comparisons, cointerventions, settings, and outcomes or measures of interest. 

The synthesis of evidence may also include a meta-analysis that pools data from included studies.  A meta-analysis will address the heterogeneity among study effects, statistical uncertainty, and the sensitivity of conclusions to changes in the protocol, assumptions, and study selection (Institute of Medicine, 2011).

Tools for the Synthesis of Evidence:

Additional tools can be found by searching the Systematic Review Toolbox (SR Toolbox).  

Report Your Findings

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) is an established guideline for reporting systematic reviews.

See the Standards and Guidelines page of this guide for other evidence synthesis reporting guidelines.