Selecting Evidence for Comparative Effectiveness Reviews (Text Version)
On September 16, 2009, Melissa McPheeters made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (913 KB).
Selecting Evidence for Comparative Effectiveness Reviews
Melissa McPheeters, PhD., MPH
Vanderbilt University Evidence-based Practice Center
- Why use selection criteria?
- What do they look like in the context of a systematic review?
- When should they be set?
- How do they affect the conduct and interpretation of the review?
CER Process Overview
- Refine key questions
- Develop analytic frameworks
Search for and select studies:
- Identify eligibility criteria
- Search for relevant studies
- Select evidence for inclusion
- Extract evidence for studies
- Construct evidence tables
Analyze and synthesize data:
- Assess quality of studies
- Assess applicability of studies
- Apply qualitative methods
- Apply qualitative methods (meta-analyses)
- Rate the strength of a body of evidence
Study Selection Criteria
- Function the same in systematic reviews as in primary research
- Should reflect the analytic framework and key questions
- Are powerful tools for widening or narrowing the scope of a review
- Provide information to determine whether reviews can be compared or combined
Types of Criteria
- Population descriptors
- Intervention descriptors
- Outcome descriptors
- Study design
- Study size
- Source of literature (including grey lit)
Some Example Criteria
- Adult, community dwelling females
- Study of a screening tool for depression
- United States only
- Hospital-based studies only
- Randomized controlled trials
- Articles published in English or German
- Review study goals
- Assess analytic framework and key questions
- Tie criteria to PICO(TS)
- Set criteria before beginning abstract review
Using broad criteria
- Can be as broad as all related studies
- Helpful for exploring "what is known"
- May result in too much literature to feasibly review, or disparate literature that can't be compared.
Using narrow criteria
- May return too little literature
- Can result in inability to answer the intended question
- Helpful in culling homogenous literature
- Can reduce size of the literature to a manageable scope
Bias in this context
- Distortion of the estimate of effect that comes from how studies are selected to be included
- Affects the applicability or "external validity" of the review itself
Bias in this context II
- For example, included studies may not have been conducted in the patient population whose care the review is intended to affect
- E.g., the use of studies of twin pregnancies in a review of preterm labor management for low risk women
- Or - selection criteria may be set to include more of a certain study type that either over or under estimates effectiveness
- What is the relevant population?
- What is the intervention of interest?
- To what exposure is the intervention being compared?
- What outcomes are relevant?
- Should time to outcome be limited?
- In what setting should the results be applicable?
- Grey literature: New technologies may have substantial grey literature, including govt. reports and trade journals
- Non-English articles may be useful for certain types of interventions (e.g. CAM)
- Study size limitations may depend on what is available
- Easier to set for observational studies
- Large bodies of efficacy/effectiveness lit may be made up of small trials
- What would you do...
- If you were asked to review the literature on transition support for adolescents with autism entering adulthood?
- Before seeing the key questions, consider the categories of criteria that we will want to apply.
- Population - condition, disease severity and stage, co-morbidities, patient demographics
- Intervention - dosage, frequency, and method of administration
- Comparator - placebo, usual care, or active control
- Outcome - health outcomes, morbidity, mortality, QoL
- Timing - Duration of follow-up
- Setting - Primary, specialty, in-patient, co-interventions
- What constitutes an adolescent? What constitutes a diagnosis of ASD?
- How is transition support defined?
- Do we compare to no transition support or directly compare types of support?
- What are the goals for adolescents with ASD as they transition to adulthood? Should they be individually focused?
- How quickly should the outcomes be apparent?
- Is transition support provided in multiple settings such as schools, clinics and the community?
What would you do with .
- A paper that was about "individuals over age 10?"
- A paper that was about an intervention for individuals with a range of developmental disabilities?
- Or conversely, about children specifically with Asperger syndrome but not other ASDs?
Example of a narrow scope
- What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?
- Efficacy: RCTs only
- Low risk: No prior preterm birth
- No. of studies: 11 studies
- Management of Overactive Bladder among Women
- OAB is a fairly difficult condition to define
- Treatments include pharmacologic, behavioral, CAM and procedural - and each area includes very different types of studies
- Its study is often combined with other types of urological conditions such as stress incontinence or prostate issues
- Set two criteria and consider the potential implications:
- Minimum study size
- Gender of study participants
- 50 at study start
- Excluded for size only: 79
- Excluded for N < 20: 36
- Excluded for N 20-29: 23
- Excluded for N 30-39: 8
- Excluded for N 40-49: 12
- Studies had to include at least 75% women
- This decision was based on expert opinion and the size and scope of the literature
- 40 studies were excluded with less than 75% women
- 27 additional studies would have been excluded had the review been limited to studies of only women
- What study designs should be included?
- Include foreign studies? Other languages? Studies conducted in specific parts of the world?
- Is the "grey" or "fugitive" literature included?
Types of Studies?
- Limit to RCTs?
- Include observational studies?
- If so, which kinds?
- What is the value of case series?
- What counts as a case series?
- Maternal-fetal Surgery Technical Brief
- Included case series with N>=2
- Only 3 of 169 studies were RCTs and 122 were case series
- Because of the relative newness of this area of research, it was important to capture data even from studies without comparison groups
- Cohorts (with comparisons)
- Case controls
- Case series
- Well-done observational studies can address issues of applicability and need for longer-term outcomes if they:
- Include more representative patient populations
- Have relevant comparators
- Report more meaningful clinical outcomes over longer timeframes
- May be a better source of information about harms
- Positive findings may be more likely to be published in high-profile journals published in English
- Therefore, to include only English may overestimate the positive effect of an intervention
- Empirically, the bias associated with limiting one's review to English only has been shown to be small
- Some content areas may be more heavily published in the foreign literature (e.g., some CAM interventions)
- NIH-OMAR State-of-the-Science conference on outcomes of cesarean delivery on maternal request
- Define the appropriate population group and comparator.
- What other limitations would you put on included literature?
- No evidence on outcomes of CDMR vs other modes of delivery
- Urgent need for actionable evidence
- Need to recognize and account for confounders
- Expand search to include proxies, weight rungs of evidence to account for confounding
- Highest rung: Trials of breech delivery, but only for maternal outcomes
- Next rung: Planned cesarean versus planned vaginal
- Lowest rung: Comparisons of maternal and neonatal outcomes from actual modes of delivery
- Selection criteria (aka inclusion/exclusion) are an essential tool for setting the scope of the review
- They should be tied to the analytic framework, key questions and PICOTS
- When properly applied, selection criteria can reduce bias and support the applicability of the review