1. Johnson, J.A. & Bootman, J.L. (1995). Drug-related morbidity and mortality: a cost-of-illness model. Archives of Internal Medicine 155(18), 1949-56.
2. Ernst, F. R. & Grizzle, A. J. (2001). Drug-related morbidity and mortality: updating the cost-of-illness model. Journal of the American Pharmaceutical Association 41(2), 192-199.
3. Referenced in RFTO 05R000075 "Evaluation of AHRQ's Pharmaceutical Outcomes Portfolio: Request for Task Order June 8, 2005."
4. Fact Sheet: Centers for Education and Research on Therapeutics. AHRQ Pub. No. 02-P025 Interim revision April 2004.
5. The exact evaluation period varies somewhat because some findings included data from before 2002 (e.g. publications, case studies).
6. Wasserman, S. and Faust, K. Social Network Analysis: Methods and Applications, Cambridge University Press, Cambridge, England, 1994.
7. This description of a CERTs project was provided by the CERTs Coordinating Center.
8. A project was considered completed when the analysis is done and the results are being presented or a manuscript is being written or published. Completion was determined by the CERT.
9. If a manuscript was published on-line within the timeframe it was included.
10. The completion dates of projects were determined from 1) the dates provided in the database or 2) associated publication dates within the range.
11. Based on primarily on discussion data.
12. The Impact of Studies funded under the Outcomes of Outcomes Pharmaceutical Research. AHRQ. October 2001. This is the perceived level of impact determined based on the information available. Level of impact: Level 1: Impact on knowledge base, future research; Level 2: Impact on policies and change agents; Level 3: Impact on clinical practice; Level 4: Impact on patient outcomes.
13. Per the Tunis and Stryer criteria for Levels of Impact.
14. Preskill, H. and Coughlan, A. Using Appreciative Inquiry in Evaluation, Number 100, Winter, 2003, Jossey Bass, San Francisco.
15. See, for example, the healthcare section. http://appreciativeinquiry.case.edu/practice/organization.cfm?sector=21 of the Web site "Appreciative Inquiry Commons".
16. Borgatti,S., Everett, M. and Freeman, L., Ucinet for Windows: Software for Social Network Analysis (Analytic Technologies, Inc., Harvard, MA, 2002).
17. Except for respondents (researchers) selected for the Case Studies. These respondents were asked if they would be comfortable with their name being openly attributed to their research that was being featured. A few seemed uncomfortable, so the discussion leader offered to have them review the write-up before final submission of the report.
18. A publication list was compiled from the Coordinating Center database of publications and from updates requested of each CERT. We did not distinguish the source of funding for a publication (i.e. core versus leveraged) due to lack of ready access to such data.
19. Garfield, E. (2006). History and meaning of the journal impact factor. Commentary. JAMA.
20. CERTs Values: Communication retrieved from http://www.certs.hhs.gov/about_certs/values.html
21. AHRQ Evidence Report/Technology Assessment No. 79. Diffusion and Dissemination of Evidence-based Cancer Control Interventions: Summary.
22. Carpenter D, Nieva V, Albaghal T, & Sorra J. Development of a Planning Tool to Guide Research Dissemination. Advances in Patient Safety: Vol. 4. https://www.ahrq.gov/downloads/pub/advances/vol4/Carpenter.pdf (Plugin Software Help)
23. Rogers, E, Diffusion of Innovations as referenced in Carpenter et al.
24. The research outputs may also include educational outputs if the publication or presentation is focused on an educational topic, however it was not feasible to discern these outputs for publications and presentations.
25. This category includes magazine articles, encyclopedia entries, and symposium publications.
26. Per the CERT Coordinating Center.
27. Note this is not the average for all CERT projects; rather it is the average publication for the projects that do have an associated publication.
28. This educational resource was created for a Portfolio grant that was described in the grant section.
30. Risk Series: Program Overview: http://www.certs.hhs.gov/programs/risk_series/index.html
31. The Impact of Studies funded under the Outcomes of Outcomes Pharmaceutical Research. AHRQ. October 2001.
32. The characterization of CERTs work was primarily based on interview data and further description of the findings beyond the titles of the projects - which can be difficult to determine. The level of impact research was also based on the review of CERTs annual reports, individual CERTs annual progress reports, and available Grantee final reports.
33. IOM Quality Chasm Report.
34. IOM Medication Errors report 2006.
35. Health Plan Employer Data and Information Set, National Committee for Quality Assurance.
36. The other important element of the MARC project was that this was added on top of an existing physician quality improvement program that had been, that we had implemented with the participation of the major managed care organizations in Denver, and as well as the state health department and other folks. So in many ways we're looking at an incremental gain of patient and public education.
37. See methods on document review for a description of the impact factor.
38. Kuruvilla S, Mays N, Pleasant A, and Walt G. (2006). Describing the impact of health research: a Research Impact Framework. Health Services Research 6:134.
39. This project was part of the project entitled CERTs Prescribing Safety Program: Overall Safety of Current Drug Use with Richard Platt as the Principal Investigator.
40. Medscape Medical News, November 18, 2005, "Inconsistent Adherence to Black Box Warnings: A Newsmaker Interview With Anita Wagner, PharmD, DPH".
41. Authors: Wagner AK, Chan KA, Dashevsky I, Raebel MA, Andrade SE, Lafata JE, Davis RL, Gurwitz JH, Soumerai SB, Platt R. FDA Drug Prescribing Warnings: Is the Black Box Half Empty or Half Full? Pharmacoepidemiol Drug Saf. 2006 Jun;15(6):369-86.
42. Lexis Nexis report was provided by a HMO Research Network staff person.
43. ISI Journal Citation Report 2005.
44. LaPointe NM, Al-Khatib SM, Kramer JM, Califf RM. Knowledge deficits related to the QT interval could affect patient safety. Ann Noninvasive Electrocardiol. 2003 Apr;8(2):157-60.
45. Al-Khatib SM, Allen LaPointe NM, Hammill BG, Chen AY, Kramer JM, Califf RM. A survey of health care practitioners' knowledge of the QT interval. Journal of General Internal Medicine 2005;20:392-396.
46. Publications III-V were identified by Duke CERTs investigators as associated/applicable publications beyond the linked publications in the CC databases.
47. The predecessor research project was identifying nutritional rickets in African American breast-fed infants.
48. Kreiter SR, Schwartz RP, Kirkman HN, Charlton PA, Calikoglu AS, & Davenport ML. Nutritional rickets in African American breast-fed infants. Journal of Pediatrics 2000;137:153-157.
49. Vitamin D Expert Panel Meeting October 11-12,2001; Atlanta, GA. Final Report.
50. Issue Briefs. These are four-page summaries of research results that highlight their social and policy relevance. They are written in easy-to-understand language with bullet points, headers, margin cut-outs and other devices to enhance delivery of the message. They are professionally written, formatted, printed and distributed to a wide, but carefully selected, audience of senators and members of congress and their staff, other politicians, key industry representatives, and other individuals who do not read scientific journals but are in a position to use the research results.
51. About CERTs: Values retrieved from http://www.certs.hhs.gov/about_certs/values.html
52. NIDA Characteristics of Centers: "Centerness", Activities, & Administrative Considerations. Retrieved from http://www.nida.nih.gov/Funding/CentersFig1.html
56. CERTs Values: Multidisciplinary retrieved from http://www.certs.hhs.gov/about_certs/values.html
54. CERTs Values: Public-Private Partnerships retrieved from http://www.certs.hhs.gov/about_certs/values.html
55. CERTs Program NCP Committee Recommendations: Partnerships internal document provided to Abt by the Coordinating Center.
56. Category includes non-profit groups, national and state councils, organizational/professional boards, and software companies.
58. "Readouts" are output from the AI exercise described in the Methods.
59. CERTs and Portfolio grant financial data were often incomplete and inconsistent. During the discussions, stakeholders offered perspectives on funding mechanisms and priorities. AHRQ expects CERTs to seek outside funding to supplement its core funding; therefore perspectives and findings on these sources of funding are described.
60. Financial data were requested of each of the CERTs directly.
63 Descriptions of these projects are provided in the Section 3.3 Educational Outputs.
62. In kind contributions were not included in this analysis due to the difficulty of quantifying them. Examples of in-kind contributions include personnel/experts such as the CERTs Steering Committee chair, planning committee members, and non-personnel contributions such as meeting space. As one example, the Duke Clinical Research Institute (the site of the Coordinating Center) in-kind contributions included faculty, staff, and meeting facilities.
63. "Others" include primarily registration fees collected from respondents of several think tank meetings and a contribution by the UNC CERT in 2001 in support of a think tank meeting.
64. Description of the Pharmaceutical Outcomes Portfolio goals in RFTO for this evaluation.
65. Retrieved from http://aspe.hhs.gov/hhsplan
66. CERTs Fact Sheet retrieved from https://www.ahrq.gov/clinic/CERTsovr.pdf
67. Excerpt from the abstract for the Risk Series Portfolio grant, retrieved from AHRQ GOLD database.
68. The research outputs may also include educational outputs if the publication or presentation is focused on an educational topic, however it was not feasible or analytically important to discern these outputs for publications and presentations.
69. Checkel, JT. It's the Process Stupid! Process Tracing in the Study of European and International Politics. Centre for European Studies University of Ohio. Working Paper No. 26, October 2005.
Return to Contents
a. Shoemaker, Jordan, Luce, Kumin, Fitzpatrick: Abt Associates, Inc.
b. Fredericks: Indiana State University
c. Weiss: University of Maryland
Return to Contents
Proceed to Appendix 1