This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.
Please go to www.ahrq.gov for current information.
Session 3: The Potential Impact of Clinical Informatics on Health Care Costs, Quality, and Safety (continued)
Cindy DiBiasi: Now let's talk about what you did in the second part.
Dr. Gretchen Berland: The second part of our study says, well,
let's say that you did land there. What we did is we picked some
of the more popular health-related Web sites that people go to according to
a number of surveys. These are surveys that, for instance, the Los Angeles
Times tells us are the most popular visited health-related Web sites. We
said, OK, for these four conditions, how comprehensive and accurate is some
of the information provided on these health Web sites?
Our approach to this part of the study was really, if you will, to design a fair test to
assess places of information on these Web sites. What we did was first
for each condition, we convened panels of patient advocates and clinical experts
for each of the conditions. We said, “Your task is to come up with
5-7 sort of 'need to know' if you will, topics and consumer-oriented
questions.” So what we really said to them, if you were a content
provider on a Web site, and you have a patient who has a question about breast
cancer, what are the basic five questions that you think should be there?
Let's say if you have a car, the engine should be there. Then we said, for
a question, we also want to know, what are answers or what are concepts that
if a consumer were to go to a Web site with a question, what should be there?
Sort of, what their bar-types of information should be there. This was part
of the test. Let me just give you an example of one of the questions
that the breast cancer expert panel came up with. The question was, “I
have a lump in my breast. What should be done to check it?” Very
The expert panel felt that these four elements below are types
of information that a consumer should find on a Web site. Breast lumps
should be brought to the attention of a physician. Mammography
and ultrasound are useful in evaluating lumps. A negative mammogram doesn't
eliminate the need for further evaluation. A persistent, non-fluid filled
breast mass felt by a physician should be biopsied. These are the types
of questions that each of the expert panels designed for each of the questions
as well as sort of the basic concept that this should all be there.
After we designed these questions for each condition, we then had two searchers
go to each of the Web sites we were studying and spend 90 minutes with the
questions looking to see if they could find the answers. So if you went
to a Web site and you searched for this information, the products of this research
were then saved and put into a notebook. All information identifying
the site was (unclear).
We then developed a standardized rating form,
a scoring system if you will, and recruited expert reviewers to evaluate the
information on two domains: coverage and accuracy. Was the information
present at all and if information was present, what was the level of accuracy?
This was a test that we wanted to make it fair and objective. Rather than
just say, “Well, do you like what you find or do you don't like
what you find?” Let me just give you an example of how we ask the
physicians and reviewers to rate coverage for breast cancer screening.
The question that was saw earlier, “I have a lump in my breast.” If
a Web site didn't mention mammography at all, it received a score of
no coverage. But if a Web site said that mammography was a way to identify
early-stage breast cancers, but it didn't tell you when the screening
should start or if there was a difference in screening for high-risk populations,
it got a score of minimal coverage. But if it mentioned that women over
the age of fifty should have the early mammogram and talked about some of the
pros and cons of mammography screening of different populations, it got a score
of more than minimal coverage.
So these are the bars that we set for the reviewers to look at the information.
Cindy DiBiasi: So what were the results in this part of
Dr. Gretchen Berland: OK, so what did we find? On English
language Web sites, you can see what this shows here that this is the percentage
of topics that the reviewers felt were handled really well. This meant
that breast cancer did the best. Overall, two thirds of the topics that
expert panelists felt should be on a Web site were covered well. Unfortunately,
childhood asthma and obesity did not do as well at all. Barely a third
of the information was covered well and on depression about half of that information
was covered well on some of the more popular Web sites.
This looks pretty good compared to what we found on Spanish sites. In
this instance, you can see that really almost the best condition was breast
cancer, which was rarely covered as well as the worst topic for English language
sites. What we don't show here is that nearly half of the topics
that the expert panel felt should be covered on a Web site about these topics
weren't covered at all. Nearly two thirds of the elements that
related to depression weren't found.
Cindy DiBiasi: It is really fascinating. This is so profound.
Dr. Gretchen Berland: There is not only a difference in the quality
of the information, but there was a difference in the quantity of information. For
instance, if you were to go to a Web site and spend an hour and a half searching
for information on breast cancer, on an English language site, on average,
you will pull about 250 pages of material off that search. If you spend
an hour and a half on a Spanish language Web site, you pull about 70 pages
There is not only a difference in quality, but there is also a
significant difference in quality in terms of, you really point out an important
point, there is a huge disparity in the quality of health information.
Cindy DiBiasi: Let's talk about the final part of
Dr. Gretchen Berland: Right. This is the really important point,
which is that let's say you can find the information. Let's
say the information is there, but the third part of our study really asks the
question, “Well, is it really going to be of any use to the population?” This
is important because we know that the few Internet survey groups and the Harris
Interactive Group reports that the Internet is becoming more and more representative
of the U.S. population. That more people who are high school educated
are going online. More Hispanics are going online.
What we did was we used widely accepted readability formulas which measure grade levels
as a function of basically word complexity in a sample of text and applied
these formulas to randomly selected passages of text in the Web sites that
we studied. This is important because most people in the United States
have graduated with a high school level education, but the actual average reading
level in the U.S. is about the eighth grade level.
We found that on average, health information is not necessarily going to be accessible
to many people. The average reading level on English language Web sites is the freshman level
in college. The lowest level reading level on the Web sites that we studied
was grade ten, which meant that for anybody who has a high school level education,
much of the information they can find may not be useful to you because you
can't read it. Spanish language sites did a little bit better. If
you are a sophomore in high school, there were two sites that were acceptable
that had a lower reading level.
Cindy DiBiasi: So what does this mean? What issues
does this raise for public policymakers?
Dr. Gretchen Berland: I think this is the first study that really
takes a slice of the technology that is very popular. I think it
means that the promise of the Internet, many of us talk about it as serving
as sort of the leveler or the Great Digital Divide and the ultimate tool that
can really improve access to care for people. I think what we found initially
is that we probably still have a little bit of ways to go.
Overall, we didn't expect to find the huge deficiency in Spanish language material.
We didn't realize there would be such a variation in quality. We
were surprised to find that there is such a variation by condition. Breast
cancer, the good news is that for breast cancer overall the information is
pretty good. We were a little disappointed to see that childhood asthma
and obesity didn't do quite so well. I think also disappointing
to us as well as the California HealthCare Foundation is that the reading levels
on many of the health-related Web sites are probably still too high if we really
talk about this tool serving as a tool that can really reach a wider, a broader
Cindy DiBiasi: It really needs to be taken down to the level
of the consumer press almost, a typical newspaper reader might be 13 years
Dr. Gretchen Berland: I mean, the New York Times I think
is written at grade 13. I think that the Web is such a fluid pool, that
what we found is we are not developing the right content. We are developing
contents for providing health information, or providing health information
that is successful to people is really hard. But the Web is a place where
you can provide information to a variety of levels, to people of different
levels of education. I also mean this is also technology where the information
doesn't necessarily have to be provided in text. Put it in video. You
can put it in audio.
I think probably the study was conducted and we
finished the study in November of last year. Since then, I know
the Web sites have modified their sites. They have put up more videos. What
we have found is that we set a bar and it probably needs more work.
Cindy DiBiasi: Great. We will be getting back to you
because I know there are going to be a lot of questions on that. In a
moment, we are going to open up the discussion for your questions. Please
recall that you can communicate your questions to us in the following ways:
if you are on the phone, just dial “14” and you will get right
through to us. Or you may fax your questions to (301) 594-0380 or
you may E-mail us your question at firstname.lastname@example.org. You may also directly
type your question in the messaging field and hit “enter.”
Please also remember that if you prefer not to use your name when you communicate
with us, that is fine, but we would like to know what State you are from and
the name of your department or organization. So please indicate that
regardless of the way in which you transmit your question.
Before going to questions, however, I would like to say a few words about
AHRQ and the User Liaison Program. The mission of AHRQ is to develop
and disseminate research-based information that will help clinicians and other
healthcare stakeholders make decisions to improve healthcare quality and promote
efficiency in the way that healthcare is delivered.
The User Liaison Program [ULP] serves as a bridge between researchers and State and local
policymakers. We not only take research information to policymakers so they are better
informed, we take the policymakers' questions back to researchers so they are aware
of the priorities. Hundreds of State and local officials participate
in ULP workshops every year. As a relatively new addition to the ULP
portfolio of products, we hope that today's Web-assisted teleconference
and the other two events in this Web-assisted teleconference series provided
a forum for discussion between our audience of policymakers and researchers
like those joining me for our discussion today.
We would appreciate any feedback you have on these teleconferences. Please E-mail
your comments to the AHRQ User Liaison Program at email@example.com. Now we are going to
go to our questions from our participants that are coming in.
The first one is to Pete. Is the public able to research a specific
doctor's records through the South Carolina System?
Pete Bailey: No. We do not believe that
the data systems are really ready for that in terms of the quality that exists
there. I don't know when that world will be or if we ever want
that world to be, but this is strictly for the provider. I think this
question will continue to (end of tape)...to make sure that we have quality
providers, but I think that a good part of healthcare is making sure that we
don't destroy the bond of trust that is between patients and their doctors. I
personally do not think that the average patient, I mean, I think we just had
a great discussion about the difficulty of understanding.
I do not think the average person would be able to have the statistical ability to
understand a lot of the issues and to be able to (unclear) like just for AIDS. It
is just a very complicated way. I hope that our world doesn't
go in that direction.
Cindy DiBiasi: One of the complexities is, aren't
physicians afraid of being compared to other physicians who treat less complicated
Pete Bailey: I had heart surgery several years
ago and the night before I began thinking if I had scores on my physician and
this guy was a B- or a B+, what would that do to me? I just don't
want that world. I want us to have a world where we make sure that these
physicians have the best quality information and that there are people monitoring
that to make sure we have that quality.
Cindy DiBiasi: Gretchen, the question you said you get most
about your study, what are the best search engines to locate medical information?
Dr. Gretchen Berland: I think that there were about 12 people
who participated in this study who were listed in the JAMA [Journal of the American Medical Association] paper and
probably every one of those twelve people has a favorite search engine and
a favorite Web site. Just to answer your question, in our study there
were really only two Web sites that statistically performed better on average.
One was for breast cancer, which was Oncolink.com, which was a site that is sponsored
by the University of Pennsylvania. The other was the National Institutes
of Mental Health site, NIMH.gov. No other sites that we studied
statistically performed better. Those are sites that the reviewers felt
covered; what it really meant that if you were to send 100 people to that Web
site, that 95 percent of the time you would know that those topics were covered
Cindy DiBiasi: However, if you are going to NIMH.gov, you
have got to be talking about mental health. You have got to be searching
for those things.
Dr. Gretchen Berland: Those were the only two sites that we found
in our study that performed better. The rest performed average and we
couldn't statistically tell if they were any better or any worse.
In terms of search engines, there were two search engines that statistically
took you to content that seems to be more relevant and one was Google and the
other was Northern Lights.
Cindy DiBiasi: Karen from the V.A. in Michigan has a question
about the study being published. We should have mentioned that the study
has been published in the Journal of the American Medical Association.
Dr. Gretchen Berland: It has been published. It has been
published in JAMA in the May 23rd issue. We have had a lot of
reprint requests for it, but you can also download it if you go to the JAMA Web
site, http://www.jama.com. You can find the paper there as well as a number of
additional tables that were posted on the Web only.
Cindy DiBiasi: Anne, a question for you. Is there
a way to query a private/public insurance payment by diagnosis?
Dr. Anne Elixhauser: Yes, indeed. That is one of the
most basic searches that you can do in HCUPnet. You can look at diagnoses
in a number of ways. You can look at them by individual ICD codes [International Classification
of Disease], by
groups of ICD codes, that you may find there is a certain series of ICD codes
that describe the condition that you are looking at.
You can look at diagnoses by what we call the Clinical Classification Software, which
is an aggregation of ICD codes that we developed here at AHRQ. Also by DRG [Diagnostic Related Group]. Then
you can look at the patient's insurance status, whether they are Medicare,
Medicaid, commercial, uninsured. It classifies patients that way. That
is indeed one of the most common searches that can be done using HCUPnet.
Cindy DiBiasi: Do find that there are the top three ways
that HCUPnet is really being used? Is there one thing that really stands
Dr. Anne Elixhauser: I think probably the most common use
of HCUPnet is to look at a very specific condition and then to get any kind
of detailed information that HCUPnet provides. For example, you can look
at a condition like depression or septicemia and you can build a fairly comprehensive
profile of what is happening in hospital care for that condition. For
depression, for example, you can look at length of stay, the charges for hospital
stay with depression, how many patients are hospitalized for depression.
Then you can break it down by all kinds of patient characteristics, comparing males
and females, comparing different age groups. Then you can look at it
by peer as a previous question just illustrated. You can look at it by
low-income vs. not low-income patients. You can compare types of hospitals
and what they do in terms of the treatment of depression. You can also
look at trends in depression over time and you can compare national, regional
and State, all those estimates, for depression on the national, regional and
State level as well.
Cindy DiBiasi: Pete, I know you are also working on a plan
to link data from a number of your State's existing health-related databases
so that this information can be better used to support policy decision-making. Can
you talk a little bit about that?
Pete Bailey: Yes. This is really an exciting
area that we are involved with. Our office is, as I mentioned earlier,
a research and statistics agency and as a neutral agency, we have been able
to pull together an enormous amount of data from different agencies. What
we have found is that, our belief in general is that a high percentage of the
people with problems in society that we are attempting to try to solve with
our programs have multiple problems that cut across specialty areas or lines
of authority or how we can use our resources.
Those problems might be health or poverty or education or training or inabilities to get
a job or alcohol and drug abuse or mental illness or family structure or child support or law
enforcement. If you look at the slide that we have up now, all of these
agencies including the private sector system, which you see down at the bottom
right hand corner. We have their data in our computer down to the individual
level and we have developed a unique tracking number where we can track people
across these systems. It enables us to link health and social services
and poverty and a whole range of issues.
What we are doing with this, of course having this is one thing and being able to do
special research studies off of it is another thing, but what we are developing is a
Web-enabled data warehouse where users must have a certificate to enter the system and that
certificate would control what they have access to. What our desires
would be for any of these agencies to be able to come into the system and be
able to take a cohort and query and see that, be able to ask as many questions
and cut across systems as possible.
In the next slide, what we are trying to show is the way we are thinking about
this, and that is, how can people access all of this data? One of our
biggest challenges has been the enormous amount of data. How can you
set up a system where people can come into it and get that kind of information? We
have set it up where they can either come in through a State agency program
or service. They can come in through special populations that we have
created, like children with special healthcare needs, either through geographic
profiles or they may be interested in a particular zip code or a block or block
We are also setting this system up so that we would be able to
do analysis at the House and Senate level which will be really interesting
to be able to do that. They could come in through some demographic group
like “Children under 6” or through healthcare utilization like
you are interested in, only diabetes and you want to look at those patients. It
allows people to track through this kind of information, which I think has
Cindy DiBiasi: Gretchen, let's just talk about some
of the research steps that you are now looking at.
Dr. Gretchen Berland: Let's go to slide eight. I think
it is important to point out that probably many people online, I suspect that,
I gave this recent talk and asked 500 people who went to the Internet to look
for health information and nearly every hand came up in the room. I
think our first step really, we just classified what you might find out there
in a certain place or time. I think, as this slide will show, that we
really don't understand yet how health information influences the doctor/patient
There is a lot of anecdotal information about patients
bringing in more health information in to their doctors from the Web that we
haven't really studied how it influences utilization. Whether patients
are asking for different things, if they are asking for inappropriate things
or more appropriate things. We haven't studied that. Probably
most importantly, we haven't studied how health information is influencing
healthcare outcome. Is it improving people's health? Are
people taking better care of themselves? Are the decisions that are occurring
as a result of health information having a better impact on their health outcome
or a worse impact?
Cindy DiBiasi: We certainly know anecdotally that patients
are walking in with reams of printouts from those to their doctors.
Dr. Gretchen Berland: Right. They are. I have a number
of colleagues who are oncologists where, even when I was training in medical
school, part of the standard medical history was, there is a whole kind of
series of things you need to ask. Now what a standard question is “Are
you going to the Internet to look for health information? If you are
going, to what Web sites are you going?” That's something
that you... often people will come in with chemotherapy regimens that they are
asking for because they found it on the Web. This is new.
Cindy DiBiasi: Do you have recommendations? Is that
part of the study or a followup to the study? Recommendations on how
consumers can use these Web sites more effectively?
Dr. Gretchen Berland: That is an important question. I think it
also points out one of our study limitations. Partly in research, you
would like your study to be able to answer all research questions and this
was just a first step. We hope that there are plenty more. Our
study did not use consumers. We sort of, we basically broke it down in
a very structured way. What you might do on the Internet and we tried
to mimic what consumers might do.
I think there needs to be more studies that look at how consumers use health
information. How consumers use the Internet. I know the National Cancer Institute has a
lot of usability testing studies where they, how do you place things in a way that it is
easier for people to use? That may mean maybe for the elderly or people who
may potentially have limited vision. So how consumers use health information
is really some of the next steps that either we will be doing or we hope that
other groups are doing.
Cindy DiBiasi: Wow. Anne, a question from Jean. She
wants to know what States are participating in HCUP.
Dr. Anne Elixhauser: I was afraid that question would come
up and I of course don't have a list. I am sure this is like the
Academy Awards and I am going to forget somebody. Like I said before,
we have got 24 participating States right now. States from all corners
of the country: Arizona, California, Colorado, Connecticut, Florida, Georgia,
New Jersey, New York, South Carolina, Washington, Wisconsin, Oregon, there
is probably more.
Cindy DiBiasi: You did them almost entirely in alphabetical
order. That's impressive. (laughing)
Dr. Anne Elixhauser: At this point, we have about 90 percent
of all hospital discharges from the Northeast and from the West. We have
got really good coverage there. We have got a little less than 50 percent
of discharges from the South and the Midwest. We are trying to really
build up our participation in those two areas.
Cindy DiBiasi: What future enhancements are you planning
Dr. Anne Elixhauser: Probably one of the biggest things
that we are going to be adding in the next six months or so is quality indicators. Pete
mentioned those a little bit earlier. A number of States have been
using these. What we did a couple of years ago is a group of us here
at AHRQ, in response as a matter of fact to the HCUP [Healthcare
Cost and Utilization Project] partners, developed what
we referred to as the HCUP Quality Indicators. These are indicators of
healthcare quality that you can look at using just hospital administrative
We have recently just released a study that Stanford University
has completed for us where they expanded and enhanced what we referred to as
the next generation of HCUP Quality Indicators. What Stanford has done
is come up with 45 measures. Things like in-hospital mortality for specific
conditions or procedures. Volume of procedures for those conditions
where there has been a demonstrated link between the number of procedures that
are done in the hospital and the outcome of care, such as in-hospital mortality.
We have some measures for utilization of procedures for procedures where there
is some question about the appropriateness of care, like C-sections and hysterectomies.
There is also a group of indicators looking at what are referred to as ambulatory
care-sensitive conditions, which are indicators that don't even look
at what is the quality of care inside the hospital, but are using hospital
data to get at potential primary care problems. Problems in the delivery
of care in the community. For example, pediatric asthma is a great example.
Most kids should be able to be treated on an outpatient basis so that they
never have to be hospitalized for their asthma. If there is a high rate of
asthma hospitalizations for children, there is something going wrong in the
primary care those kids are receiving. Other measures like that would
be diabetes complications. Again, people with diabetes should be able
to be treated outpatient without getting admitted for short-term diabetes complications
like coma and ketoacidosis.
Stanford is continuing development. They are going to be releasing these later in the
year, what we refer to as Patient Safety Indicators or Complications of Care that can be
looked at using again hospital administrative data. We are going to be applying these
quality indicators to our HCUP data. That will then be put on the Internet so
that individual States can have national and regional comparisons. How
does my State compare on these indicators with the nation as a whole or with
other States in my region? Or individual hospitals can compare their
performance with specific peer groups. Other teaching hospitals in the
Northeast, for example. That sort of thing.
Cindy DiBiasi: Let's talk a little bit about the HIPAA [Health Insurance Portability and Accountability Act] requirements. Pete, let's start with you. What are the implications
of those requirements on your initiatives?
Pete Bailey: We are certainly scared of what
they might do to us and I certainly am no expert in that area. The way
we are doing our best, we have done this before with HIPAA. We developed
a unique tracking number. We used identifying information to develop
this tracking number, but when data come to us, we separate the identifying
data and the statistical data. We use the identifying data to develop
the tracking number and then we link that tracking number over to the statistical
If we get data on you from four or five different agencies
or sources, you would have that same tracking number. We never put any
information together about you on anything that would identify you. Not
in our office, not anywhere, because I think that does break the law. The
other thing that we think we are in pretty good shape on is that the work we
are doing, we have laws that require the data and that also I think (unclear).
Cindy DiBiasi: Anne?
Dr. Anne Elixhauser: I guess HIPAA affects us in a couple
of ways. One is the standardization of healthcare information. That
actually is going to help HCUP a lot because we spend an awful lot of resources
right now just standardizing the data that we receive from the States, converting
it into a uniform format so that we and other researchers can do analysis with
it. Once HIPAA standards are in place, we can basically tend to other,
more interesting things. Instead of just standardizing the data, we can
do more research and spend our resources in other ways.
With respect to privacy, there are a couple of ways that HIPAA affects us and a couple of
ways that we have sort of dealt with it already. HCUPnet, for example,
does not use, it is not built on the data itself. We don't have
patient-level data that underlies HCUPnet. HCUPnet is based on aggregate
statistics. It is based on principles of data warehousing where we have
these huge data cubes that we have basically compiled all possible combinations
of data and made that available through HCUPnet.
One of the reasons why HCUPnet works so quickly, if you had to actually do the principal
analysis right on the data itself, it could take 45 seconds to a minute to get your
results, which would be deadly on the Internet. It just wouldn't
hold on to anybody. Because we have these data cubes and because everything
is aggregated already, whenever you do a search, most of the searches come
up pretty quickly.
The main reason that we did that was because we didn't
want to have the patient-level data on the Internet so that a hacker could
get in and actually get access to the discharge-level data. The side
benefit of it was the speed that it provided. The HIPAA privacy issues
are affecting us just in terms of I think we're mostly just concerned
about the extent to which data can be shared with us. There are a number
of people who are on the HCUP staff who are working very actively with a number
of the States who are concerned about this, trying to understand the HIPAA
There are a couple of people on our staff who have actually
helped to draft some guidances that hopefully will be released as part of explanations
for the HIPAA regulations so that it explains exactly what HIPAA means in terms
of sharing data with us and then us making data available to researchers.
Return to Contents
Proceed to Next Section