Skip to Main Content

Systematic Reviews and Other Expert Reviews

#systematicreviews #sysrev

Developing Search Strategies in Multiple Databases

Systematic reviews require searches in multiple databases to help ensure comprehensiveness. When developing search strategies in different databases, keep in mind the following tips and best practices:

  • Use the controlled vocabulary for each database; controlled vocabularies include MeSH in PubMed and Emtree in Embase. 
  • Adapt search syntax (field tags, phrase searching, etc.) for each database.
  • Check whether proximity operators are available and will help with keyword searching.
  • Use validated search filters as appropriate for each database. See the "Using Search Filters" box below for more information.
  • Decide on your choice of databases before you begin the searching process.
    • If you find that one database is finding many more results than other databases and that these results are off-target, you should not abandon this database, but rather look for reasons why this is happening. Welch informationists have experience troubleshooting these kinds of issues.

Using Search Filters

Search Filters
  • Filters whose recall and precision have been validated, such as those listed under the adjacent tabs, are most commonly used to locate specific study types (e.g., RCTs) or to exclude animal studies.
  • However, rarely will a review of observational studies include a study filter.  
  • The PubMed RCT filters developed by the Cochrane Collaboration (provided under the adjacent tab) are considered the gold standards for identifying randomized controlled trials.
Limits
  • Avoid limiting by language, since this may introduce bias into your search results.
  • Limit by date only if this is dictated by the content of the review (e.g., the date a drug, product, or medical test was first introduced).
  • Consider carefully whether to include a limit (e.g., age) as a concept in your search strategy or as part of your inclusion/exclusion criteria. Consider testing whether including a limit in your search will cause you to miss relevant results.
  • Be prepared to justify any search limits to manuscript reviewers. Limits should not merely be convenient ways to reduce the number of results. Limits should be reasonable and appropriate regardless of the search results.

The following filters were created and validated by the Cochrane Collaboration and are listed on the Cochrane Work Review Group website:

Sensitivity-Maximizing Version (2008 revision) - Maximizes recall

(randomized controlled trial[pt] OR controlled clinical trial[pt] OR randomized[tiab] OR placebo[tiab] OR drug therapy[sh] OR randomly[tiab] OR trial[tiab] OR groups[tiab] NOT (animals [mh] NOT humans [mh]))

Link to this filter in PubMed.

Sensitivity- and Precision-Maximizing Version (2008 revision) - Provides a balance of recall and precision

(randomized controlled trial[pt] OR controlled clinical trial[pt] OR randomized[tiab] OR placebo[tiab] OR clinical trials as topic[mesh:noexp] OR randomly[tiab] OR trial[ti] NOT (animals[mh] NOT humans [mh]))

Link to this filter in PubMed.

A validated Embase RCT filter is in development.  Until a validated filter is available, the Cochrane Handbook (Part 2, Section 6.4.11.2 - Search filters for identifying randomized trials in EMBASE), recommends two sources for filters. Please consult each source to decide which filter is most appropriate for your topic. 

Lefebvre et al. Filter

'crossover procedure':de OR 'double-blind procedure':de OR 'randomized controlled trial':de OR  'single-blind procedure':de OR (random* OR  factorial* OR crossover* OR cross NEXT/1 over* OR placebo* OR doubl* NEAR/1 blind* OR singl* NEAR/1 blind* OR assign* OR allocat* OR volunteer*):de,ab,ti

Source: Lefebvre C, Eisinga A, McDonald S, Paul N. Enhancing access to reports of randomized trials published world-wide - the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library. Emerging Themes in Epidemiology 2008; 5:13.


Wong et al. Filters

Sensitivity Maximizing Strategy
random*:ab,ti OR (clinical NEXT/1 trial*):de,ab,ti OR  'health care quality'/exp

Specificity Maximizing Strategy
(double NEXT/1 blind*):de,ab,ti OR placebo*:ab,ti OR blind*:ab,ti

Best Optimization of Sensitivity and Specificity
random*:ab,ti OR placebo*:de,ab,ti OR (double NEXT/1 blind*):ab,ti

Source: Wong SS, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically sound treatment studies in EMBASE. Journal of the Medical Library Association 2006; 94(1): 41-47.

Note that the filters provided in Lefebvre et al.and Wong et al. are for use in Embase on the Ovid platform. The Cochrane Work Group has translated these filters for use in the Embase.com platform, which is the platform you have access to at Hopkins.

Rarely, a review of observational studies will include a study filter. In these cases, check for validated filters before building your own.

Several organizations maintain lists of filters.

Note: If you choose to use one of the Qualitative Study filters for PubMed, you will need to update the MeSH term. As of December 2015, “questionnaires”[mesh] is now “surveys and questionnaires”[mesh].

Evaluating the Quality of Search Strategies

Why Evaluate Search Quality?

It's important to evaluate the quality of your search strategies for a number of reasons, including:

  • Helping you decide when you can stop optimizing draft searches
  • Giving you confidence in the comprehensiveness of the evidence identified
  • Providing evidence to reviewers and readers about the thoroughness of your searching techniques
Evaluating Quality Using Gold-Standard Articles
  • Test to see if your search strategies are capturing gold-standard articles in the field. These are also known as index articles.
  • Keep in mind that not every article that you have pre-identified as relevant to your research will be an index article for your search. An index article must capture all of the main ideas of your research question.
Evaluating Quality Using the PRESS Checklist

The PRESS Checklist provides guidelines for the peer review of search strategies. It includes the following question prompts.

  • Is the search question translated well into search concepts?
  • Are there any mistakes in the use of Boolean or proximity operators?
  • Are any important subject headings (i.e. controlled vocabulary terms) missing or have any irrelevant ones been included?
  • Are any natural language terms or spelling variants missing, or have any irrelevant ones been included? Is truncation used optimally?
  • Does the search strategy have any spelling mistakes, system syntax errors, or wrong line numbers?
  • Do any of the limits used seem unwarranted or are any potentially helpful limits missing?
  • Has the search strategy been adapted for each database to be searched?

Search Updates

In order to capture new, relevant documents during a systematic review project, a search update may be warranted. A search update involves running the original searches again, typically with date limits to capture documents since the last search.

When performing a search update, keep the following in mind:

  • A search update may involve adding or updating some terms if controlled vocabularies have changed during the course of the project.
  • Updating searches for databases without good date-limiting functionality may be challenging. Welch informationists have experience addressing these challenges.
  • Systematic review guidelines, journals, and funders may have specific requirements for search updates, so it's important to be aware of these when planning your project.
    • A key guideline for systematic reviews, the Cochrane Handbook for Systematic Reviews of Interventions, states that review teams should, "Rerun or update searches for all relevant sources within 12 months before publication of the review or review update, and screen the results for potentially eligible studies."

Documenting Search Strategies

Developing comprehensive search strategies for a systematic review takes time and multiple attempts. There are benefits to documenting these multiple attempts when developing search strategies. There are also requirements for documenting the final search strategies when reporting your findings in a publication.

How Documentation Helps You and Others
  • Help you make decisions when optimizing draft search strategies
  • Help others to evaluate the quality of your final search strategies and reproduce them
Documentation That Helps When Optimizing Draft Search Strategies
  • Search concepts that could be added to focus results
  • Search concepts that could be removed to broaden results
  • Search terms that could be added to improve comprehensiveness
  • Search terms that could be removed to avoid off-target results
  • The number of pre-identified, gold-standard articles found
Documentation Required When Reporting Final Search Strategies

The PRISMA guidelines for reporting systematic reviews lays out the requirements for reporting the search strategies of all databases searched for a systematic review. These elements include:

  • Any limits applied (e.g., date, language, etc.) with justifications
  • Published search filters used (cited, with any modifications noted)
  • Validation approaches (e.g., through the use of pre-identified, gold standard articles)
  • Peer review (e.g., through the PRESS checklist)