Appendix II. Identifying Measures
Care Coordination Measures Atlas
Main Indicator Sources
- Literature Search. A measure-specific care coordination search was conducted to identify published literature related to the development, validation, and testing of measures of care coordination. The search strategy is outlined below.
- Care Coordination EPC Report. As part of a previously published care coordination report ("Closing the Quality Gap: A Critical Appraisal of Quality Improvement Strategies"; Volume 7: Care Coordination), background research and a systematic review identified care coordination indicators in published studies.
- Panelist Calls. A series of panel calls were held in order to obtain information regarding additional measures of care coordination and ongoing research and development in the field. Panel participants had backgrounds ranging from research and evaluation in care coordination to clinical practice. For a list of panel participants, please go to Appendix III: Advisory Group Participants.
- NQF Draft Report. NQF evaluated a list of 77 candidate measures and recommended a set of preferred practices across five domains of care coordination: 1) health care home, 2) proactive care plan, 3) communication, 4) information systems, and 5) transitions. The final report was released in October, 2010, shortly before completion of the Atlas.15
Literature Review Search Strategy
The final measure search was performed on July 13, 2010, using the following strategy:
[ ("(("healthcare" or "healthcare" or care) adj3 (coordinat* or "co-ordinat*" or integrat*)).tw.") And ("(rated or rating or indicator* or measure* or valid* or reliab* or outcome* or model* or scale* or subscale* or questionnaire*).tw. or methods.fs. or exp Questionnaires/") ] Not [ ("exp geographic locations/ not exp united states/") ]
The search was limited to English language publications. Details of the search strategy development are included in the box below.
In addition, publications by known key researchers involved in care coordination measurement were also searched. Bibliographies of particularly relevant included references were also reviewed for any further sources of information.
We compared our search strategy to RAND's ACOVE-3 search strategy post-hoc, and we found no additional terms, phrases, or combinations that were not captured in the strategy outline above.
Details of Search Strategy Development
With the help of a research librarian, a literature search was conducted using Ovid MEDLINE® In-Process & Other Non-Indexed Citations and Ovid MEDLINE® 1950 to Present. Several sets of search terms were used in combination to net articles describing care coordination measures and measurement strategies. Search sets 1-7 (shown below) represent the concept of 'care coordination'. Search sets 8-10 represent the concept of 'measures'. Search sets 11 and 12 represent the concept of 'outpatient' and 'medical home'. Search set 13 represents any citation indexed with a non-United States country subject heading and is used with the Boolean operator "NOT" to narrow the size of other search sets. Search set 14 is a high-precision title search using only the most relevant terms to "catch" obviously relevant citations the other searches might have missed.
Searches using combinations of the above sets were conducted. Searches were checked for article inclusion compared to a list of 10 highly relevant articles. The most effective search strategy was chosen based on inclusion rate and the total number of search results. The team determined that the final search strategy should yield no more than 4000 results, while simultaneously including as many of the 10 "test articles" as possible. The final search strategy used was: ((7 and 10) not 13) and eng.la. A search of the database through April 5, 2010, using this search set yielded 3306 publications and included 8 of the 10 test articles. The measure search was updated on July 13, 2010, to capture any additional measures indexed in MEDLINE® after the original search. The updated search yielded 8 new measures from among 142 new publications.
Search Set 1: "exp "Continuity of Patient Care"/" (10856 results)
10 Relevant Articles:
Measures for inclusion in the Atlas were identified in two steps. First, a list of potential measures was compiled from the search strategies outlined above, which yielded a total of 3448 unique measure sources. Measure sources were included if they featured any relation to measurement or evaluation of care coordination or of any of the care coordination domains included within our measurement framework, with an emphasis on specific instruments or measures. Although the ambulatory setting is the focus of this project, we did not exclude sources discussing measurement of care coordination in nonambulatory settings. Validity, testing, or feasibility of measures were not considered during this review phase.
A single reviewer compiled the potential measures list after reviewing titles and abstracts of all search results. A second reviewer provided input on measure sources for which inclusion was unclear and a decision was made through discussion.
Based on this preliminary review, 149 potential measure sources were identified to consider for inclusion in the Atlas. It is important to note that in some cases multiple sources related to a single measure, and in other cases a single source discussed multiple measures.
In the second step, we reviewed the full text of all articles on the potential measure list and made decisions about whether they should be included within the Atlas. Measures were excluded if, in the opinion of the reviewer, they did not meet all of the following criteria:
- Clear relevance to care coordination or at least one of the care coordination measurement framework domains. Measures that did not include at least one instrument item that mapped to at least one framework domain were not included. Measures that contained only 1 or 2 minimally relevant items within a large instrument unrelated to care coordination were also excluded.
- A clearly defined and reproducible measure yielding quantitative data. Examples of evaluations that did not meet this criterion were interview guides, focus group reports, or free-response questionnaires yielding textual data that required content analysis; quality improvement guides designed to walk users through a process of self-evaluation without yielding measurable data; and evaluations of specific programs or interventions tailored to the subject of study in such a way as to make use in any other situation very difficult without major modification.
- Information available demonstrating some valid measurement properties or that the measure was developed in association with a logic model that has evidence of causal linkages between the activities measured and outcomes desired. Measures that underwent testing and were shown to have poor validity or reliability were not included in the Atlas.
In many cases, additional sources were consulted to address the testing criteria. When the decision about whether to include a measure seemed unclear, the primary reviewer consulted with additional team members and a decision was made through discussion.
Of the 149 potential measure sources identified, 70 were excluded. Of these, 31 were excluded due to lack of relevance (criterion 1); 34 were excluded because they were not a clearly defined, quantitative measure (criterion 2); and 38 were excluded due to unknown or poor validation or testing (criterion 3). Thirty-five potential measure sources met more than one exclusion criteria. In addition, 7 measures sources were excluded because we could not identify information necessary to assess suitability for inclusion in the Atlas. Our attempts to obtain the missing information from the developers of these measures were unsuccessful at the time of publication. A further 18 measure sources were not unique; that is, multiple sources pertained to a single measure. These sources were used to create the profiles but did not themselves contribute a unique measure.
In all, we include 61 measures in the Atlas, which are detailed in 78 profiles. The number of profiles is greater than the number of measures because for measures with multiple versions, we created separate profiles for versions with substantially different question items. In instances where the only difference between versions was a minor wording change to reflect a different population, setting, or year, we created just one profile to represent all versions.
Although we attempted to identify as many potential measures of care coordination as possible through our various search strategies, we relied primarily on published instruments available in the public domain. Instruments not published in journals were identified chiefly through suggestions from our review panels. This method omits an unknown number of potential measures that were not published in the literature, not identified by our search terms, or were not recommended for review by our advisory panels. When a potential measure of care coordination was reported in the literature without including the measure instrument, we contacted the article author to request a copy of it. We were also limited in our ability to provide information on the feasibility and cost of using measures by what was reported in the literature; few studies describe these aspects of measurement.