This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.
Please go to www.ahrq.gov for current information.
AQA Invitational Meeting Summary
Report of the Performance Measurement Workgroup
Kevin Weiss, American College of Physicians
Kevin Weiss opened his
remarks by noting that although the Performance Measurement Workgroup does not
have a perfect process, it does have one that is robust, transparent, and
self-effacing. He briefly reviewed past accomplishments, including the
- Approving the AQA
Parameters for Selecting Ambulatory Care Performance Measures.
- Approving a 26-measure AQA
ambulatory care starter set.
- Beginning to enlarge the
measure set to include cardiology, cardiac surgery, and patient experience
in ambulatory care (ACAHPS) measures (for a total of 86 measures, counting
ACAHPS as a single measure).
- Approving the Guide for
the Selection of Performance Measures for Medical Subspecialty Care.
- Anticipating that
registries will be a part of the discussion and approving Principles in the Use of Registries for
Enhancing Quality of Care Through Performance Measurement.
- Approving the AQA
Principles of "Efficiency" Measures.
Next, Weiss outlined his workgroup's
and approve the proposed list of prioritized conditions/procedures for
the cost-of-care measures discussion paper.
and approve the quality metrics for emergency medicine, gastroesophageal
reflux disease (GERD), hematology, and oncology.
other topics under development and get feedback on how to prioritize a
long and robust set of ideas.
Opening the discussion on
cost of care, Weiss said that in 2007 the workgroup hoped to do the following:
- Finalize and gain approval
for a cost-of-care principles document.
- Work with measure
developers to see the priority list of conditions for cost-of-care work turned
- Further refine the
discussion around total cost-of-care measures.
He then briefed participants
on how the Prioritized List of Cost-of-Care Conditions had been
developed. Weiss said that the Cost-of-Care Workgroup had approved with an "overwhelming
consensus" a proposed starter set of 20 conditions and recommended that the
list be referred to the AQA Performance Measurement Workgroup for review and
approval. The Performance Measurement Workgroup then reviewed the conditions.
Using a modified Delphi process, the workgroup engaged in two rounds of voting
and discussion to arrive at the selected seven prioritized cost-of-care
The prioritized conditions address the following:
- Acute myocardial infarction.
- Congestive heart failure.
- Coronary artery disease.
- Low back pain.
Weiss noted that six of the
seven conditions were approved with overwhelming majority support. He said the
key criterion for inclusion was that there had to be quality metrics associated
with each condition so that cost and quality could be measured. Weiss
noted, however, that there was some discussion within the workgroup as to
whether the measures were too narrow (three address cardiovascular care) and
also as to whether there were quality metrics for depression.
The workgroup addressed the
seventh condition, low back pain, separately. Although low back pain did not
have quality measures endorsed by the AQA (and thus no quality metrics in
place), the workgroup ultimately decided to include it on the priority list.
Weiss said that most workgroup members approved the decision, although there
were two "no" votes and a few abstentions.
Following Weiss' remarks, there was considerable discussion about whether to include low back pain. A number of people argued that low back pain should be taken off the list because of the lack of quality measures.
Others spoke in favor of keeping low back pain on the prioritized list. Several participants noted that low back pain impacts a wide range of patients. They also noted that there is enormous variation in the care and treatment of low back pain across a number of specialties. Several others stressed that low back pain is an enormous problem for the purchaser community. One person noted that low back pain is generating significant costs and quality concerns. She added that if the AQA did not address the condition, then various health plans and employers will start to study the issue on their own.
Finally, two people pointed out that metrics are available on low back pain. One said that a set of low back pain measures is in operation in Ontario and that this program can be studied to see if it is feasible. Another said low back pain can be measured on the basis of outcomes, such as activities of daily living.
There was also discussion of other items included on—and left off—the prioritized list. One participant noted that there are no robust sets of measures for depression, asthma, or some of the other conditions on the prioritized list and that there is a lot of uncertainty over whether we can develop measures of cost of care and whether they can be appropriately applied to individual physicians.
One participant wondered why cancer is not on the prioritized list. In response, Weiss indicated that, although three types of cancer (prostate, breast, and colon) had been on the initial candidate list of conditions, the quality metrics were not as crisp as they were for those conditions that did make the priority list.
Several participants expressed concern that 24 months was too long a timeframe for developing cost-of-care measures and that a more aggressive timeline was needed. In response, Weiss said that moving faster was really resource-dependent. He added that the Quality Alliance Steering Committee and the NQF's Efficiency Workgroup are also addressing cost of care.
A representative from the NQF noted that one of the reasons the priority list of conditions was expanded to seven items (from an initial plan to target five) is that the NQF is already looking at two of the measures. She said that the NQF process is studying what aspects of performance (both quality and cost) should be assessed given the evidence base and gaps, and will also consider safety and patient engagement on decisionmaking. She said that the NQF hopes to complete the process for at least the first two conditions within 1 year.
One person asked for clarification as to whether the goal was to look at diseases across settings or at the whole patient with a disease or with a disease across the entire continuum of care. Weiss replied that the workgroup has not yet gotten to that level of specificity but that he suspected the NQF workgroup would address these questions. He added that the discussion at this meeting was focused on disease but that the measure developers could later focus on the details.
Finally, someone noted that quality and cost-of-care measures must go hand in hand and that the first question to be asked about cost-of-care measures would be "How do we know what the impact on quality is going to be?" In response, Weiss stressed that the AQA was not being asked to vote at this meeting on cost-of-care measures but only on prioritization of areas for development of cost measures.
Before any votes were taken,
there was discussion about whether to vote on the priority conditions en bloc
or one at a time. One person suggested a motion to remove low back pain from
the list. Another suggested a separate vote on low back pain. Another argued for
an en bloc vote, saying that none of the conditions should be separated out.
There was also considerable
discussion about how to manage the voting process and who would be eligible to
vote. Votes for this meeting were by recorded show of hands (where necessary)
and based on the principle of one vote per organization.
Before the debate over how
to vote on low back pain, Carolyn Clancy raised the issue of the AQA's voting procedures.
She said that the AQA Steering Committee would bring a recommendation on voting
procedures for future meetings before the AQA at its next meeting.
endorse the Prioritized List of Cost-of-Care Conditions, excluding low
Result: The motion was adopted with near
Motion: To include low back pain within the Prioritized List of Cost-of-Care Conditions.
Result: The motion was adopted, 53-24, with 4 abstentions.
Before moving to the next agenda item, one person stressed that the goal of the AQA has been to move performance measures forward as quickly as possible, pointing out that AQA must be able to move an agenda and give every stakeholder confidence that there is a clear path forward with a defined timeline. He added that AQA needs to figure out how to get the funding necessary to move an agenda that is precise, is clearly defined, and contains very clear timelines.
Regarding timelines, Weiss noted that AQA is moving quickly—there hadn't even been any discussion about cost of care a year ago—but to move forward even faster, more resources would be needed.
Next, Weiss introduced the draft Cost-of-Care Measurement Set document. The purpose of the document is to
define the objectives of cost-of-care measures and the key components to help ensure
reliable measurement. The document notes that the Performance Measurement Group
recognizes the need for:
- Measures that may be related to specific conditions or procedures.
- A composite measure of the relative cost-of-care performance of a physician's total practice.
The document also details
anticipated uses for cost-of-care measures, including the following:
- Performance education and
- Clinical performance
- Performance incentivization.
- Transparency and consumer
- Iterative improvement of
In addition, the document
- Measurement systems and
the need for transparency.
- The need to present cost-of-care
measures, wherever possible, as average costs compared with expected costs.
- Principles and procedures
for prioritizing and developing condition- or procedure-specific starter
The paper will be brought
forward for a vote at the next AQA meeting.
Following Weiss' remarks,
two participants said they were glad that language addressing feedback and
integrity ("there should be a formal feedback and correction mechanism so that
errors uncovered by physicians, plans, and other analysts can contribute to
improving the evaluation system") had been included.
Another person strongly endorsed the document, saying that it would provide great guidance to employers, who are looking for tools like this.
Proposed New Quality Measures for Adoption
Kevin Weiss discussed new
quality measures for:
- Emergency medicine.
- Gastroesophageal reflux disease (GERD).
Weiss said that, when the workgroup
was considering the measure sets, two crosscutting issues came up:
- The issue of entry or
competency measures versus high-performance measures.
- The issue of exclusions.
Weiss elaborated on the first issue, saying that it was important to talk about the spectrum of measurement and that no threshold has been put on any measures yet, as there was not yet a framework for doing that. Weiss asked participants to set aside the issue to vote on the measures today, saying that he would seek to address the matter either through the AQA Performance Measurement Workgroup or in partnership with another pertinent forum.
- Electrocardiogram performed for non-traumatic chest pain.
- Aspirin at arrival for acute myocardial infarction (AMI).
- Electrocardiogram performed for syncope.
- Vital signs for community-acquired bacterial pneumonia.
- Assessment of oxygen saturation for community-acquired bacterial pneumonia.
- Assessment of mental status for community-acquired bacterial pneumonia.
- Empiric antibiotic for community-acquired bacterial pneumonia.
Quality Improvement Only:
- Fibrinolytic therapy ordered within 20 minutes of electrocardiogram performed for AMI.
- Care coordination for percutaneous coronary intervention for AMI.
In introducing the proposed
emergency medicine measures, a participant representing the American College of Emergency Physicians noted that the measures are currently under
consideration in the NQF.
Motion: To approve proposed emergency medicine measures No. 1-7.
Result: The motion was adopted with near unanimity.
Gastroesophageal Reflux Disease (GERD)
- Assessment for alarm symptoms.
- Upper endoscopy for patients with alarm symptoms.
- Biopsy for Barrett's esophagus.
- Barium swallow—Inappropriate use.
Quality Improvement Only:
- Medication therapy—Assessment of GERD symptoms.
In introducing the proposed GERD measures, a participant representing the American Gastroenterological
Association noted that the measures have been through the Physician Consortium
for Performance Measurement process. He said they were subsequently submitted
to the NQF and have been reviewed by an NQF technical advisory group.
Weiss indicated that there
had been considerable discussion within the Performance Measurement Workgroup about
whether the GERD measures should be assigned a specialty. He said there was
also a lot of discussion around measure No. 1 (assessment for alarm symptoms). Ultimately,
all but a few people voted to report out the entire measure set.
Following Weiss' remarks,
one participant raised the issue of primary care. He said that the issue of how
expansive the measure set will be needs to be addressed. What will primary care
physicians be accountable for and need to report on? Another participant expressed
concern that the notation regarding "adequate data sources only if new codes
are developed" would preclude the use of administrative data.
approve the four proposed GERD accountability measures.
Result: The motion was adopted with
- Myelodysplastic Syndrome and acute leukemias—Baseline cytogenetic testing performed on bone marrow.
- Myelodysplastic Syndrome—Documentation of iron stores in patients receiving erythropoietin therapy.
- Multiple myeloma—Treatment with bisphosphonates.
- Chronic lymphocytic leukemia—Baseline flow cytometry.
In introducing the proposed
hematology measures, a participant representing the American Society of
Hematology outlined the development process. He said that the measures have gone
through the Consortium process and are ready to be submitted to the NQF.
Weiss noted that the
workgroup had discussed whether or not these measures were too entry-level and whether
they fit in well with standards of care. He said the workgroup ultimately
endorsed the hematology measures with few "no" votes or abstentions.
approve the four proposed hematology measures.
Result: The motion was adopted with
- Cancer staging documented in the medical record.
- Hormonal therapy for stage IC-III estrogen receptor/progesterone receptor positive breast cancer.
- Chemotherapy for stage III colon cancer patients.
- Plan for chemotherapy documented before chemotherapy administered.
- Radiation therapy for invasive breast cancer patients who have undergone breast conserving surgery.
In introducing the proposed
oncology measures, a participant representing the American Society of Clinical
Oncology (ASCO) said her organization has a long history of developing measures.
She added that ASCO had a library of more than 50 measures tested in 100
oncology practices around the United States.
Weiss acknowledged that
ASCO's path to developing measures is different from the multistakeholder
process that the AQA encourages. He said that, in voting to bring the oncology
measures to the full AQA, the workgroup understood that the Consortium would
take up the measures (due to happen in late February) and that the measures
would then be submitted to the NQF. In addition, the amount of predevelopment
work and the substantial and robust vetting process that had taken place within
ASCO convinced the vast majority of workgroup members to vote "yes." He said measures
No. 2 through No. 5 were voted out of the workgroup with very few "no" votes.
One person raised concerns
about approving the oncology measures, saying that having a consensus process
with multiple stakeholders at the table is crucial. Another person noted that
measures No. 2 and No. 3 address what should be prescribed. He suggested that,
where appropriate, measures should look not only at what is prescribed but also
at whether a prescription has been filled. Although this second element isn't
completely under a physician's control, it is the responsibility of physicians
to engage their patients to comply.
Another participant noted
that measures No. 2, No. 3, and No. 5 have made their way through the NQF
process. Speaking in support of approval, he indicated that the oncology
community would like to have something to report on. A participant representing
the NQF clarified that the three measures had not yet been endorsed by the NQF.
She added that the NQF has put them forward as potential system-level measures
(not clinician-level measures).
approve proposed oncology measures No. 2-5.
Result: The motion was approved 42-22, with 6 abstentions.
Turning to measure No. 1, Weiss indicated that it had been the most controversial within the workgroup for two reasons. First, it was not developed using the same process as the other measures; rather, it was added later. Second, there were concerns that the measure was too basic.
One participant wondered whether measure No. 1 was a performance measure and had been created merely so the other ones could be implemented. In response, the ASCO representative said that the data suggest that not all physicians are currently documenting the staging information. Another participant said that, although the measure is very basic, it identifies a quality gap. A third participant spoke in opposition to the measure because it had not been vetted through a multistakeholder process.
approve proposed oncology measure No. 1.
Result: The motion failed.
Work in Progress
Next, Weiss turned to the
workgroup's extensive to-do list for 2007. He reminded people to review the Cost-of-Care Measurement Set document and provide any feedback as the workgroup
intended to bring the document forward for a vote at the next AQA meeting. He
also asked participants to review the draft AQA Principles for Appropriateness
Measures document (designed to provide guidance to measure developers on
how best to construct implementable appropriateness measures that will meet
multistakeholder needs). In addition, Weiss noted that the workgroup is developing a test measure designation as well as principles for AQA test measures (and what that designation might mean), adding that they would like to field-test measures before the measures must be reported on publicly.
Weiss also highlighted other
items on the workgroup's 2007 agenda:
- Continuing to review
measures submitted by medical and surgical specialty societies.
- Pursuing measurement on the
prioritized list of cost-of-care conditions.
- Pursuing measurement on composite
measures for total cost of care.
- Integrating the
workgroup's efficiency measures work with that of the Quality Alliance
Steering Committee and the NQF.
- Discussing when measures
can be studied on an individual vs. physician group level.
- Developing principles
around measure implementation/data sourcing (including consideration of principles around exclusions).
- Beginning to discuss
prioritization of focus on specialty measure development.
- Considering developing
principles of use of measures along the competency/high-performance
spectrum (and setting the criteria to allow the AQA to determine and even
possibly tag where on the spectrum a measure falls).
- Considering principles in
the use of measurement as related to special populations.
- Considering quality and
- Developing principles for
composite quality measures.
- Encouraging the
development of ACAHPS measures for surgery and other population needs
Opening the discussion on prioritizing the agenda, one participant suggested that the place to start was with the things we know are going to happen and cross-correlate those with what physicians can tolerate. He also suggested laying out where the gaps are. In response, Weiss
said that there was a pretty clear picture of how the development process was
proceeding within both the Consortium and the NQF. The missing piece, he said, is
"measures that are okay but not really what we need." He added that there was
also a need to prioritize the focus on specialty measure development.
One participant expressed concern about setting priorities. What aren't we going to do? Everyone in a multistakeholder process has different priorities.
Weiss said that one approach
would be to look at and get a better understanding of where some of this work
is already being undertaken within the broader health care community.
suggested that the workgroup ought to first map out how the various initiatives
might be staged. She pointed out that the AQA offers a unique opportunity to
have a conversation among multiple stakeholders about how to think about
quality going forward. This follows a model that has worked well up until now (measures
developed by specialty societies, brought to the Consortium, and then brought
to the NQF).
The same participant also touched on the issue of the spectrum of competency and whether or not this should be left to boards and licensing bodies to address.
One participant raised a
question regarding individual vs. physician group analysis. Will CMS be
committed to collecting these data by provider identifier number (and not just
by tax ID number)? In response, a CMS official indicated that the agency was
committed to addressing the issue and trying to get some answers.
suggested that the AQA needs a mechanism for identifying a measure's developer,
knowing whether the measure has been endorsed by the NQF, and knowing whether the measure has been
approved by the AQA. He pointed out that it was very important for those in the field to figure out the status of a measure and that this was not possible today. The same participant also raised two other issues:
- The importance of understanding the disease burden of physicians (which is different from the specialty).
- The need to map out measures that are appropriate for physicians' practices.
One participant spoke of the need to have a strong emphasis on more meaningful, outcome-oriented, high-performance measures. He added that he strongly supported the view that the workgroup should focus on populuations at risk and known gaps in care.
Another participant pointed out the need to start to coordinate multiple initiatives/groups and to define the strategic needs of the patient in health care. Patients want to be able to find a quality physician. In addition, he said that more strategic networking is needed outside the AQA process in order to create a learning network to start to close gaps in quality care.
The same participant warned against measure overload and stressed the danger of creating a lot of measures (because of the incentives for reporting) that don't align in strategic ways. Referencing the agenda item on prioritizing focus on specialty measure development, he said it was important to develop a measure that addresses how to deliver care to patients regardless of setting—rather than measuring how physicians do this—and that the effort must include key initiatives in primary care so that the strategic needs of patients are met without putting an undue burden on physicians.
Several other participants also stressed the need for a patient-centric approach. One participant noted the need to drive high-performance measures that are meaningful for the consumer and the need to prioritize efforts around health disparities.
The next step, according to one participant, is to embed the AQA's work into everyday work.
Summing up the discussion, Weiss said the message that each of the agenda items is important was clear. He said it was also clear that the workgroup needed to place a little more emphasis on a few agenda items, and he promised to bring those back for discussion at the next AQA meeting.