Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Selecting Evidence for Comparative Effectiveness Reviews (Text Version)

Slide presentation from the AHRQ 2009 conference

On September 16, 2009, Melissa McPheeters made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (913 KB).

Slide 1

Slide 1. Selecting Evidence for Comparative Effectiveness Reviews

Selecting Evidence for Comparative Effectiveness Reviews

Melissa McPheeters, PhD., MPH
Associate Director,
Vanderbilt University Evidence-based Practice Center


Slide 2

Slide 2. Overview


  • Why use selection criteria?
  • What do they look like in the context of a systematic review?
  • When should they be set?
  • How do they affect the conduct and interpretation of the review?


Slide 3

Slide 3. CER Process Overview


CER Process Overview

Prepare topic

  • Refine key questions
  • Develop analytic frameworks

Search for and select studies:

  • Identify eligibility criteria
  • Search for relevant studies
  • Select evidence for inclusion

Abstract data:

  • Extract evidence for studies
  • Construct evidence tables

Analyze and synthesize data:

  • Assess quality of studies
  • Assess applicability of studies
  • Apply qualitative methods
  • Apply qualitative methods (meta-analyses)
  • Rate the strength of a body of evidence

Present findings


Slide 4

Slide 4. Study Selection Criteria

Study Selection Criteria

  • Function the same in systematic reviews as in primary research
  • Should reflect the analytic framework and key questions
  • Are powerful tools for widening or narrowing the scope of a review
  • Provide information to determine whether reviews can be compared or combined


Slide 5

Slide 5. Types of Criteria

Types of Criteria

  • Population descriptors
  • Intervention descriptors
  • Outcome descriptors
  • Study design
  • Study size
  • Source of literature (including grey lit)


Slide 6

Slide 6. Some Example Criteria

Some Example Criteria

  • Adult, community dwelling females
  • Study of a screening tool for depression
  • United States only
  • Hospital-based studies only
  • N>200
  • Randomized controlled trials
  • Articles published in English or German


Slide 7

Slide 7. Selecting Criteria

Selecting Criteria

  • Review study goals
  • Assess analytic framework and key questions
  • Tie criteria to PICO(TS)
  • Set criteria before beginning abstract review


Slide 8

Slide 8. Using broad criteria

Using broad criteria

  • Can be as broad as all related studies
  • Helpful for exploring "what is known"
  • May result in too much literature to feasibly review, or disparate literature that can't be compared.


Slide 9

Slide 9. Using narrow criteria

Using narrow criteria

  • May return too little literature
  • Can result in inability to answer the intended question
  • Helpful in culling homogenous literature
  • Can reduce size of the literature to a manageable scope


Slide 10

Slide 10. Bias in this context

Bias in this context

  • Distortion of the estimate of effect that comes from how studies are selected to be included
  • Affects the applicability or "external validity" of the review itself


Slide 11

Slide 11. Bias in this context (cont.)

Bias in this context II

  • For example, included studies may not have been conducted in the patient population whose care the review is intended to affect
    • E.g., the use of studies of twin pregnancies in a review of preterm labor management for low risk women
  • Or - selection criteria may be set to include more of a certain study type that either over or under estimates effectiveness


Slide 12

Slide 12. Basic Questions

Basic Questions

  • What is the relevant population?
  • What is the intervention of interest?
  • To what exposure is the intervention being compared?
  • What outcomes are relevant?
  • Should time to outcome be limited?
  • In what setting should the results be applicable?


Slide 13

Slide 13. Considerations


  • Grey literature: New technologies may have substantial grey literature, including govt. reports and trade journals
  • Non-English articles may be useful for certain types of interventions (e.g. CAM)
  • Study size limitations may depend on what is available
    • Easier to set for observational studies
    • Large bodies of efficacy/effectiveness lit may be made up of small trials


Slide 14

Slide 14. Exercise 1

Exercise 1

  • What would you do...
    • If you were asked to review the literature on transition support for adolescents with autism entering adulthood?
    • Before seeing the key questions, consider the categories of criteria that we will want to apply.


Slide 15

Slide 15. Apply PICOTS


  • Population - condition, disease severity and stage, co-morbidities, patient demographics
  • Intervention - dosage, frequency, and method of administration
  • Comparator - placebo, usual care, or active control
  • Outcome - health outcomes, morbidity, mortality, QoL
  • Timing - Duration of follow-up
  • Setting - Primary, specialty, in-patient, co-interventions


Slide 16

Slide 16. PICOTS


  • Population
  • Intervention
  • Comparators
  • Outcomes:
  • Timing
  • Setting
  • What constitutes an adolescent? What constitutes a diagnosis of ASD?
  • How is transition support defined?
  • Do we compare to no transition support or directly compare types of support?
  • What are the goals for adolescents with ASD as they transition to adulthood? Should they be individually focused?
  • How quickly should the outcomes be apparent?
  • Is transition support provided in multiple settings such as schools, clinics and the community?


Slide 17

Slide 17. What would you do with . . ..

What would you do with .

  • A paper that was about "individuals over age 10?"
  • A paper that was about an intervention for individuals with a range of developmental disabilities?
  • Or conversely, about children specifically with Asperger syndrome but not other ASDs?


Slide 18

Slide 18. Example of a narrow scope

Example of a narrow scope

  • What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?


Slide 19

Slide 19. Selection Criteria

Selection Criteria

  • Efficacy: RCTs only
  • Low risk: No prior preterm birth
  • No. of studies: 11 studies


Slide 20

Slide 20. OAB Study

OAB Study

  • Management of Overactive Bladder among Women
  • Considerations:
    • OAB is a fairly difficult condition to define
    • Treatments include pharmacologic, behavioral, CAM and procedural - and each area includes very different types of studies
    • Its study is often combined with other types of urological conditions such as stress incontinence or prostate issues


Slide 21

Slide 21. Exercise 2

Exercise 2

  • Set two criteria and consider the potential implications:
    • Minimum study size
    • Gender of study participants


Slide 22

Slide 22. Study Size

Study Size

  • 50 at study start
  • Implications:
    • Excluded for size only: 79
    • Excluded for N < 20: 36
    • Excluded for N 20-29: 23
    • Excluded for N 30-39: 8
    • Excluded for N 40-49: 12


Slide 23

Slide 23. Gender


  • Studies had to include at least 75% women
  • This decision was based on expert opinion and the size and scope of the literature
  • 40 studies were excluded with less than 75% women
  • 27 additional studies would have been excluded had the review been limited to studies of only women


Slide 24

Slide 24. Other Considerations

Other Considerations

  • What study designs should be included?
  • Include foreign studies? Other languages? Studies conducted in specific parts of the world?
  • Is the "grey" or "fugitive" literature included?


Slide 25

Slide 25. Types of Studies?

Types of Studies?

  • Limit to RCTs?
  • Include observational studies?
  • If so, which kinds?
  • What is the value of case series?
  • What counts as a case series?


Slide 26

Slide 26. Example 3

Example 3

  • Maternal-fetal Surgery Technical Brief
    • Included case series with N>=2
    • Only 3 of 169 studies were RCTs and 122 were case series
    • Because of the relative newness of this area of research, it was important to capture data even from studies without comparison groups


Slide 27

Slide 27. Observational Studies

Observational Studies

  • Types
    • Cohorts (with comparisons)
    • Case controls
    • Case series
    • Registries/databases


Slide 28

Slide 28. Observational Studies

Observational Studies

  • Well-done observational studies can address issues of applicability and need for longer-term outcomes if they:
    • Include more representative patient populations
    • Have relevant comparators
    • Report more meaningful clinical outcomes over longer timeframes
    • May be a better source of information about harms


Slide 29

Slide 29. Foreign Literature

Foreign Literature

  • Positive findings may be more likely to be published in high-profile journals published in English
  • Therefore, to include only English may overestimate the positive effect of an intervention
  • Empirically, the bias associated with limiting one's review to English only has been shown to be small
  • Some content areas may be more heavily published in the foreign literature (e.g., some CAM interventions)


Slide 30

Slide 30. Cesarean Delivery

Cesarean Delivery

  • NIH-OMAR State-of-the-Science conference on outcomes of cesarean delivery on maternal request


Slide 31

Slide 31. Exercise 3

Exercise 3

  • Define the appropriate population group and comparator.
  • What other limitations would you put on included literature?


Slide 32

Slide 32. The challenge

The challenge

  • No evidence on outcomes of CDMR vs other modes of delivery
  • Urgent need for actionable evidence
  • Need to recognize and account for confounders


Slide 33

Slide 33. Solution


  • Expand search to include proxies, weight rungs of evidence to account for confounding
    • Highest rung: Trials of breech delivery, but only for maternal outcomes
    • Next rung: Planned cesarean versus planned vaginal
    • Lowest rung: Comparisons of maternal and neonatal outcomes from actual modes of delivery


Slide 34

Slide 34. Summary


  • Selection criteria (aka inclusion/exclusion) are an essential tool for setting the scope of the review
  • They should be tied to the analytic framework, key questions and PICOTS
  • When properly applied, selection criteria can reduce bias and support the applicability of the review
Current as of December 2009
Internet Citation: Selecting Evidence for Comparative Effectiveness Reviews (Text Version). December 2009. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care