Your browser doesn't support JavaScript. Please upgrade to a modern browser or enable JavaScript in your existing browser.
Skip Navigation U.S. Department of Health and Human Services
Agency for Healthcare Research Quality
Archive print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Appropriate Drug Use and Prescription Drug Programs

Evaluation of Benefit Programs


Stephen B. Soumerai, Sc.D., Professor, Department of Ambulatory Care and Prevention, Harvard University, Boston, MA

Stephen Soumerai explained the risks and benefits of evaluation techniques and provided examples of different study designs that can best support pharmacy program goals and initiatives.

Soumerai emphasized that poor evaluation design can be misleading, and that five potential factors can taint the internal validity of any study. Observed effects of studies may be the result of changes in:

  • History.
  • Subject maturation.
  • Instrumentation.
  • Statistical regression.
  • Selection of experimental groups.

When evaluating studies for use in program design, it is important to look for these five potential threats to internal validity to understand the extent of the study's usefulness.

Soumerai gave examples of weak uncontrolled study designs and alerted participants to be wary of studies and reports that use them. All of the following study designs do not protect against the five major threats to internal validity. Soumerai presented them on a continuum ranging from the weakest design model to the strongest design model:

  • Post-Test-Only Design: Based only on observations taken after intervention. Gives no indication of historical factors that may alter results.
  • Post-Test-Only Design with Non-equivalent Groups: This yields results gathered from comparing post-test observations of a treatment group given an intervention with a comparison group given no intervention. This design also gives no indication of historical factors that may affect results.
  • Pre-Test/Post-Test Design: This design yields results gathered from comparing observations before and after intervention, yet it allows for no comparison with observations of a group in the absence of intervention.

He explained that some study designs are more effective in protecting against threats to internal validity. These include:

  • Non-equivalent Control Group Design: This design compares similar groups both before and after intervention. In this type of design, the most important factor is the comparability of the two groups. It is most desirable to select them to be as similar as possible.
  • Time-Series Design: In this design, multiple observations (ideally six to seven) are taken both before and after intervention. In a time series, it is possible to determine background trends by taking observations before intervention.

Soumerai explained that inadequately controlled studies of pharmaceutical policies often can produce misleading conclusions. He also points out that medication use and health outcomes frequently change, even in the absence of any programs or policies.

Soumerai stressed that, particularly when promoted by special interests, highly publicized, poorly controlled studies have the capacity to make large impacts on public policy. He cited a March 1996 article in the American Journal of Managed Care that attempted to show that "restrictive formularies...may be counterproductive and actually increase patient use of healthcare services, in turn increasing overall healthcare costs." Using a graph, he then demonstrated that the number of articles and negative reports about health maintenance organizations and formularies, as well as the number of formulary laws enacted, increased dramatically after the publication of the article.

Soumerai explained the benefits to using longitudinal models. Longitudinal models both increase statistical power in quasi-experimental studies and use information on trends, as well as provide graphical evidence (visible rather than statistical) by incorporating multiple pre- and post-test observations to demonstrate the impact of an intervention. A longitudinal design is important because it allows policymakers to see the impact of given interventions that would not be visible in a design that only reveals trends of the aggregate population.

Soumerai used the example of a study done on the impact of a Triplicate Prescription Policy (TPP) for benzodiazepines instated in New York State in 1989 to give an example of a policy that had significant effects. The policy's intention was to reduce the abuse and cost of benzodiazepines. The study's intention was to determine whether the policy was effective in reducing abuse and cost, yet maintaining appropriate use. Soumerai demonstrated that the policy did achieve its intended effects. Also, he explained that the time-series and comparison group designs can detect effects of physician surveillance on:

  • Target and substitute drugs.
  • Blunting of secular increases in highly effective drugs.
  • Access to new medication use.
  • Access to essential medications in vulnerable populations.

In explaining another series of studies, Soumerai discussed how to evaluate the impacts of planned interventions. He also demonstrated how time-series designs can show positive effects of successive medication quality improvement interventions and the lack of effects of unproductive approaches.

In summary, Soumerai concluded that longitudinal data allow for strong quasi-experimental designs, provide more valid results and can show visible effects that are almost always significant. When conducting evaluations, Soumerai encourages the creative use of comparison series by using different groups, such as:

  • Unexposed comparison populations.
  • High-risk subgroups.
  • Market-share indicators.
  • Unintended outcomes.


Horn, Susan D., PhD; Sharkey, P, PhD; Tracy, D., Ph.D.; Horn, C; James, B; Goodwin, F., M.D. Intended and Unintended Consequences of HMO Cost-Containment Strategies: Results from the Managed Care Outcomes Project. American Journal of Managed Care 1996 Mar;2:253-64.

Soumerai SB, Ross-Deegan D, Fortess EE, Abelson J. A Critical Analysis of Studies of State Drug Reimbursement Policies: Research in Need of Discipline. Millbank Q 1993;217-52.

Soumerai SB, McLaughlin TJ, Ross-Deegan D, Casteris CS, Bollini P. Effects of limiting Medicaid drug reimbursement benefits on the use of psychotropic agents and acute mental health services by patients with schizophrenia. N Eng J Med 1994 Sept 8;331:650-5.

Current as of August 2002

Previous Section Previous Section         Contents                        

Internet Citation:

Appropriate Drug Use and Prescription Drug Programs: Adding Value by Improving Quality. Workshop Brief, November 5-7, 2001, User Liaison Program. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.

AHRQ Advancing Excellence in Health Care