Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality www.ahrq.gov
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk factors of chronic diseases (Text Version)

Slide presentation from the AHRQ 2009 conference.

On September 15, 2009, Tatyana Shamliyan made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (265 KB).


Slide 1

Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk factors of chronic diseases

Minnesota EPC
Clinical Epidemiology Methods Centre, Ottawa Health Research Institute, Ottawa
Tufts University Medical Center, Boston
RTI International - University of North Carolina, Chapel Hill
Blue Cross and Blue Shield Association, Chicago
Centers for Disease Control and Prevention, Atlanta
Southern California EPC; RAND Corporation, Santa Monica
German Institute for Quality and Efficiency in Health Care, K�ln, Germany
Johns Hopkins University, Baltimore

 

Slide 2

Analytical tasks

Analytical tasks included 4 steps:

  • Step I- Systematic literature review:
  • Step II- Development of two checklists based on the results from literature review
  • Step III- Collaboration with the participating experts:
  • Step IV- Finalizing the checklists

 

Slide 3

Therapeutic vs. non therapeutic studies

PICOTS Non therapeutic studies Non randomized studies
Population General population Patients, health care recipients
Exposure (interventions, independent variables) Risk factors that are independent of health care (allocation irrelevant) Treatments allocation by investigators or health care professionals
Comparators Levels of risk Another trtm, usual care, standard care, previous trtm
Outcomes Incidence/prevalence of chronic conditions Morbidity, mortality, quality of life, adverse events after treatments
Time Length of exposure Length of trtm, followup off the trtm
Settings General population Health care settings
Comparability (selection bias) Not relevant Flaw
Intention-to treat Not relevant Flaw

 

Slide 4

Systematic literature review

  • Systematic reviews and meta-analyses of observational nontherapeutic studies published in English in core clinical journals (145)
  • Published tools (scales or checklists) for quality assessment of observational studies (96)
  • Annotated bibliography of the methodological literature to identify biases and strategies to reduce bias in observational studies

 

Slide 5

Development of Checklists

  • Separate evaluation of external and internal validity
  • Exhaustive range of criteria
  • Discriminating reporting and methodological quality
  • Predefined major and minor flaws
  • Study and hypotheses level analyses
  • Automated reports
  • Templates for the protocols of quality evaluations with predefined research specific quality standards
  • Manual, help files, instructions

 

Slide 6

Collaboration with participating experts

  • Criteria evaluation: face and content validity
  • Pilot reliability testing (10 randomly selected articles): overall, by topic, by article, by domains of external and internal validity, and by quality component
  • Discriminant validity: testing hypotheses of detected differences in quality and reporting vs. methodological quality

 

Slide 7

Finalizing checklists

  • Checklist for studies of incidence or prevalence of chronic diseases (MORE)
  • Checklist for studies of risk factors of chronic diseases (MEVORECH)
  • Synthesis of evidence from the studies with major flaws:
    • Level A exclusion from synthesis
    • Level B full review and quality abstraction
    • Level C separate limited synthesis if major flaws detected

 

Slide 8

Quality criteria

Incidence or prevalence studies

Aim of study
Study design

External Validity

  1. Sampling of the subjects by the investigators
  2. Assessment of sampling bias
  3. Estimation of sampling bias
  4. Exclusion rate from the analysis
  5. Sampling bias is addressed in the analysis
  6. Subject flow
Internal Validity
  1. Source to measure outcomes
  2. Definition of outcomes
  3. Measurements of outcomes
  4. Outcomes in race, ethnic, age, or gender subpopulations
  5. Reporting of outcomes

Risk factors studies

Aim of study
Objectives
Study design

External Validity

  1. Sampling of the subjects by the investigators
  2. Assessment of sampling bias
  3. Estimation of sampling bias
  4. Exclusion rate from the analysis
  5. Sampling bias is addressed in the analysis
  6. Subject flow
Internal Validity
  1. Source to measure outcomes
  2. Definition of outcomes
  3. Measurements of outcomes
  4. Definition of the exposure
  5. Measurements of the exposure
  6. Confounding factors
  7. Loss of followup
  8. Masking of exposure status
  9. Statistical analysis
  10. Appropriateness of statistical models
  11. Reporting of tested hypothesis
  12. Precision of the estimates
  13. Sample size justification

 

Slide 9

Major flaws

External validity

  • Sampling in clinics for incidence in the general population;
  • Sampling of controls from different population as cases;
  • Response rate <40%;
  • Exclusion of >10% of eligible subjects from the analyses
  • Reasons to exclude from the analyses differ for exposed and not exposed
Internal validity
  • Severity of the outcome can be relevant but not assessed in the study;
  • Non valid methods were obtained to measure the outcomes, exposure, or confounding factors;
  • Different methods to measure exposure (risk factors, independent variable) in cases and controls;
  • No strategies to reduce bias

 

Slide 10

Minor Flaws

External validity

  • Random sampling restricted to geographic area
  • Convenient sampling
  • The authors did not assess sampling bias
  • Sampling bias was not addressed in the analysis
  • Sampling of controls from health care related sources
Internal validity
  • Outcomes proxy reported
  • Obtained from medical records (data mining) or administrative database
  • Inter-methods validation
  • Subjective judgment of reliability
  • Intensity/dose of exposure can be relevant but not assessed in the study

 

Slide 11

Example of the report

Article: _________________  
Evaluator: _______________  
External Validity  
Not Reported  
Addressing sampling bias Not reported
Sampling: General population based Not reported
Major flaw  
Exclusion rate from the analysis >10%
Internal Validity  
Major flaw  
Definition of incidence/prevalence: Severity Can be relevant but not assessed in the study
Minor  
Definition of incidence/prevalence: Frequency of symptoms Can be relevant but not assessed in the study

 

Slide 12

Conclusions

  • We propose two checklists for transparent and standardized quality assessment in systematic reviews and evidence-based reports
  • Reasonable validity
  • Poor random overall and good component reliability in a pilot testing of the articles from different topics

 

Slide 13

Future Research with Quality Assessment Using Proposed Checklists

  • Protocols of systematic reviews of nontherapeutic observational studies with justified definitions of research specific quality standards and preplanned reliability testing
  • Synthesis of evidence incorporating quality assessment (sensitivity of the results to quality)
  • The evaluation of the level of evidence from several observational nontherapeutic studies
  • Quality assessment of primary studies should improve quality of systematic reviews and evidence-based reports
Current as of December 2009
Internet Citation: Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk factors of chronic diseases (Text Version). December 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://archive.ahrq.gov/news/events/conference/2009/shamliyan/index.html

 

The information on this page is archived and provided for reference purposes only.

 

AHRQ Advancing Excellence in Health Care