This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.
Please go to www.ahrq.gov for current information.
Health Care Quality
Reliability of current physician report cards questioned
A recent study supported by the Agency for Health Care Policy and Research questions whether the current generation of report cards on how well individual doctors manage specific health conditions can really help people select doctors or help doctors improve their own performance. A growing number of health plans and hospitals write report cards on their doctors.
The study looked at how doctors in three types of medical practices in different parts of the country managed patients with type II diabetes—a disease the researchers selected because of its prevalence in primary care practice and because the way doctors manage diabetes can affect outcomes in their patients.
The researchers found that report cards, or physician profiles, for diabetes were unable to reliably detect true practice differences among doctors at the three medical practices studied: a large, West Coast staff-model health maintenance organization, an urban medical school teaching clinic located in the Midwest, and a group of private-practice physicians in New England. Differences in how the physicians managed their patients' diabetes—their practice styles—contributed only 4 percent, at the most, to the overall variance in their patients' hospitalization, office visit, laboratory use, and blood sugar level control rates. The difficulty in using these outcomes to evaluate physician performance was due, in large part, to the relatively small number of patients with diabetes managed by each doctor.
Sheldon Greenfield, M.D., a coauthor of the study and chairperson of the Diabetes Quality Improvement Project—a collaborative effort by the American Diabetes Association, National Committee for Quality Assurance, Foundation for Accountability, and the U.S. Health Care Financing Administration—says the findings send a wake-up call to health plans and hospitals evaluating their physicians and to individual doctors.
Dr. Greenfield points out that these findings do not mean that physicians should not be accountable for patients' outcomes, or that attempts to evaluate the medical and quality-of-life outcomes of care are futile because the average doctor doesn't see enough patients for a specific condition to make analysis meaningful. Instead, the findings suggest that health plans and hospitals need to take a better look at the science they're using to do report cards, he concludes.
According to Dr. Greenfield, who is with the New England Medical Center in Boston, each of the physicians in the study would have had to manage over 100 diabetic patients for the researchers to detect any meaningful differences among them. Apart from hypertension, it is difficult to imagine that there would be enough cases per primary care physician to construct disease-specific profiles for almost any other chronic condition, notes Dr. Greenfield, who added that doctors sometimes receive evaluations based on as few as four patients. The science does not support such reckless use of numbers for judging physicians. More accurate judgments could be made if physicians were evaluated in groups for the purpose of treating specific diseases.
The lead author of the study, Timothy P. Hofer, M.D., of the VA Ann Arbor Healthcare System and the University of Michigan, said another problem with the current method of profiling individual doctors is that the process can backfire if it is linked to powerful incentives not to treat patients whose conditions are hard to manage because they may fail to follow instructions or for other reasons and who incur high costs. The easiest way a doctor can avoid getting a bad report card is to drop problematic patients. According to Dr. Hofer and his colleagues, those implementing such profiling systems would be as much to blame as the doctors who deny care to the most vulnerable people needing medical help.
Sherrie H. Kaplan, Ph.D., a co-author of the study and an expert on patient satisfaction notes that at least 40 or 50 patients per doctor are needed to accurately evaluate satisfaction with an individual physician's care. The current study did not report findings about the evaluation of patient satisfaction, but the issue was addressed in the AHCPR-funded Type II Diabetes Patient Outcomes Research Team Study (AHCPR grant HS06665), a 5-year research project from which the current study is derived. Dr. Kaplan, who is with the New England Medical Center, and the other authors, were members of the study team, which was under the direction of Dr. Greenfield.
Funding for the current study was also provided by the Veterans Affairs Health Services Research and Development Service—a component of the Department of Veterans Affairs. Details are in "The unreliability of individual physician 'report cards' for assessing the costs and quality of care of a chronic disease," in the June 9, 1999, issue of the Journal of the American Medical Association, pp. 2098-2105.
Return to Contents
Proceed to Next Article