Summary of the Presentations (continued 2)
Expanding Research and Evaluation Designs to Improve the Science Base for Health Care and Public Health Quality Improvement Symposium
Session II. Research Evaluation Designs for QII at the Health "Plan" Level
Case: Expanding and Testing VA Collaborative Care Models for Depression
Joseph Francis Jr., M.D., M.P.H. (Chair/Moderator)
Associate Director, Office of Research and Development, Department of Veterans Affairs (VA)
Translating Initiatives in Depression into Effective Solutions: Regional Expansion Project
Lisa V. Rubenstein, M.D., M.S.P.H.
Professor of Medicine, VA Greater Los Angeles Healthcare System and UCLA
Senior Scientist, RAND
Dr. Rubenstein intended to help participants understand this field, which is in development, and how it may or may not apply to areas outside of depression. She proposed that there is an evolution of designs that goes along with the evolution of goals in going from efficacy to effectiveness to quality improvement to routine care. Dr. Rubenstein profiled a series of studies culminating in the Regional expansion of the Translating Initiatives in Depression into Effective Solutions (ReTIDES) study. This is an ongoing project that aims to help practices implement evidence-based treatments for depression care.
Depression efficacy research based on classic randomized trials shows that both antidepressants and short-term manualized psychotherapy are effective. Descriptive studies of the quality of depression care nationally identified cases of low quality of care, disparities, and variations in routine practice. Initial randomized provider behavior change interventions to improve depression care quality focused on knowledge-related barriers through clinician education, screening and feedback, and computer reminders. These single-component interventions, however, did not improve quality of care for this condition. Finally, a series of provider behavior change interventions using multi-component models identified the effectiveness of collaborative care for depression. Collaborative care is a multicomponent model, similar to the Chronic Care Model, that fills the gap between primary care and mental health specialty care using a care manager, who sometimes is a nurse. Collaborative care also activates and educates patients for self-care. Initially, these studies randomized patients and were designed similarly to classic trials. Studies randomized at the patient level showed effectiveness across all major demographic groups, including minorities, adolescents, and the elderly. In addition, to test the effectiveness of implementing collaborative care organizational changes in practices, as would be required to disseminate the effective models, investigators randomized practices (cluster randomized designs) and evaluated the effects of collaborative care on consenting patients in experimental practices. These studies showed effectiveness and cost-effectiveness in large and small, rural and urban, managed care, and other types of practices. Finally, a series of randomized studies evaluated how practices could use quality improvement (QI) methods to self-adopt improved depression care and improve practice-wide performance. In these designs, outcomes are measured across representative patients with depression attending the experimental or control practices, independently of their participation in improved depression care - similarly to the way HEDIS (Health Plan Employer Data and Information Set) or other performance measures are carried out.
These studies showed that without evidence-based tools, practices do not impact depression care performance. Incorporating tools from previous studies is associated with perceptible, but modest impacts. Even with support for using evidence-based tools (currently available on at least four Web sites originating from randomized studies), only some practices (those with features fostering success, such as support from mental health and primary care leadership) succeed in adopting collaborative care. Additional work adapting collaborative care, as carried out in research studies, to the needs of routine practice in particular health care systems and settings appears to be necessary for successful dissemination of collaborative care, and for assuring that the disseminated model is implemented in a way that achieves impact on practice performance.
The Translating Initiatives in Depression into Effective Solutions (TIDES) project was a nonrandomized study in which performance reports helped guide PDSA (plan, do, study, act) cycles. The clinical results of the study have been very good. However, the TIDES study found that a "top down" approach to change is only effective in the VA when the "bottom up" side has already been built. The study found there was no handoff of VA-adapted collaborative care, designed through the TIDES QI process by three VA regional networks (Veterans Integrated Service Networks or VISNs), from research to a clinical entity. They concluded that they would have to go further than they had thought to make that handoff happen by interacting with many different government entities with which the VISNs are connected.
The goal of ReTIDES is to expand TIDES to a new VISN and more medical centers, providing the basis for national implementation including a fully developed business case. Regional expansion of TIDES is being evaluated with semistructured stakeholder interviews with an emphasis on performance-measure-based evaluation. This study is being done with a non-equivalent control group design with pre- and post-test measures at multiple time intervals.
Threats to the current design of ReTIDES include not being able to achieve the scope of an intervention that would affect the performance measures as well as all the usual threats to nonrandomized designs. For the study's purpose, a nonrandomized design was better than randomization. The previous randomized trials present a strong evidence base, and there would be a low level of gain from one more randomized trial compared to the gain from learning about and fostering system implementation. Randomization would further restrict their ability to study the naturalistic process of implementation. For example, there are only a limited number of comparable primary care practices in any regional network, accounting for size, rural versus urban location, academic affiliation, and existing depression care organization. Randomizing within strata of comparable practices would significantly limit management choice regarding how to roll out improved depression care at a regional level. Furthermore, randomization reinforces the idea that the implementation is research, not quality improvement, allowing participants to think of the implementation as temporary and not to think about all of the routine administrative policies, procedures, and resource allocation decisions needed to sustain collaborative care on an ongoing basis.
The ReTIDES approach has been adopted into the national VA primary care strategic plan and the hope is that this approach will be adopted into routine care throughout the VA over the next three to five years.
Improving Quality: A Steeplechase of Sorts? The Necessity of Science for Practice and Science for the Public
Junius Gonzales, M.D.
Acting Director, Division of Services and Intervention Research, National Institute of Mental Health
According to a December 6, 2003, BMJ report about the disconnect between research and practice and between policy and practice, there is a need to reconnect funders, users, researchers, and those who might benefit from research. Users and researchers agreed that no one was taking responsibility for disseminating findings and supporting application of findings. Dr. Gonzales commented on randomization versus ecologically relevant results. Dr. Rubenstein matches pairs of sites based on organizational and patient data, and Dr. Gonzales raised the issue of whether there might be contextual, dynamic, or process issues or events that would mean that sites actually were not a great match.
A rigorous examination of process data, as well as strategies for theorizing based on process data, would help make the business case as well as inform the designs of future QI studies. It is important to address balancing accuracy, generality, and simplicity. Clinicians want to know how the results from a particular RCT apply to the patient sitting in front of them who, for example, is outside the age group examined in the study and has co-existing conditions.
A number of challenges and opportunities exist for any QI study. Basic questions revolve around generalizability to other settings and populations. How collaborative care works has not been deconstructed; thus, we do not know about the potency of interactions between different elements of collaborative care. Also, no one knows what mechanisms, such as continuity of care or patient trust, make the model work. Business managers want something better, faster, and cheaper, thus they are focused on wanting to know what the "active ingredient" is that makes collaborative care work.
What we need in the area of QI research is to identify and understand modifiable mechanisms of change, and not just to describe patterns of change. We need new ways to address implementation science, such as taxonomies of near/far, serial, and expert transfer, that increase the potential for local adaptation rather than being limited to a "one size fits all" approach. On the funding side, it would be valuable to invest in the research infrastructure for "real world" settings. Sometimes the needs are very basic, such as computers for data collection. The National Institute of Mental Health (NIMH) has some funding mechanisms outside the usual grant schedule which allow researchers to move quickly to seize research opportunities on "natural experiments" as they happen.
Finally, Dr. Gonzales noted that the big issue for the improvement in care seen in Dr. Rubenstein's research is sustainability. In other words, will the changes stick?
A participant asked about the issues of contamination and floor and ceiling effects. In addition, he noted that when control groups show improvement, some people describe this as a contamination effect. He considers that pejorative and makes it likely that one will miss the opportunity to find out why the controls also improved. Perhaps one reason it was difficult to see improvement is that most people were doing well already, thus there would be a ceiling effect on improvement. To avoid that, one would have to look at the degree of improvement as a function of the baseline rate. Dr. Rubenstein responded that this is related to the issue of taking advantage of one's results, and is relevant to randomized trial findings as well. What one person would consider contamination, another would consider spread. At the patient level, in Dr. Rubenstein's studies, there is control for baseline sickness.
A participant asked why "deconstruction" of collaborative care is needed when it seems clear that the deconstructed components are not working. Dr. Gonzales replied that not all aspects of collaborative care were proven effective or ineffective. The package is transferable, but we do not always know why it works for certain groups but not others. Also, there are different models of collaborative care. The reason that elements of the collaborative care package need to be deconstructed is that practices lack funding for the whole package in a collaborative care model. Dr. Gonzales feels that the active ingredients still are not known. Dr. Rubenstein added that they are deconstructing the model in terms of taking apart its operational components in order to make the business case. For example, patients must be assessed, thus the research group is examining what happens when a patient is assessed. Her group also is attempting to test innovations in this model performed at individual sites, while trying to ensure that the model is not watered down. They want to keep the core model very tight while still being able to examine innovations.
Another participant commented that the placebo effect, as well as the Hawthorne effect, are poorly understood in cluster randomized trials. Blinding is impossible in many of these trials. Innovative designs, such as randomization in which the control group is unaware they are in a trial, or Zelen randomization, should be considered.
A participant asked about the business case for QI initiatives. What is needed to examine the business case is not a shift in the study design, but an interest in a range of variables, such as economic variables, that are not typically included in cluster randomized trials. Dr. Rubenstein responded that one can get much of the information needed for the business case from classical study designs in terms of cost effectiveness and how patients flow through the system. However, one has to look past randomized trials to truly capture the system costs, which is what the business managers are interested in. One has to let the system operate freely in order to capture things such as how cost shifts from primary care to mental health care. Dr. Rubenstein noted that it is not easy for researchers and business managers to interact given their different vocabularies. A participant noted the problem with the business case is that it depends on context (e.g., patient location, revenue, etc.) and that the level of care will vary with these circumstances.
Another participant commented that when considering the business case, one must examine who is paying. For purchasers, outcome measures should be included that focus on staff "presenteeism" and absenteeism. In areas such as depression care, there is evidence of a return of dollars from improved labor outcomes.