Session 3: The Potential Impact of Clinical Informatics on Health Care Costs, Quality, and Safety (continued)

Cindy DiBiasi: Now let's talk about what you did in the second part.

Dr. Gretchen Berland: The second part of our study says, well, let's say that you did land there. What we did is we picked some of the more popular health-related Web sites that people go to according to a number of surveys. These are surveys that, for instance, the Los Angeles Times tells us are the most popular visited health-related Web sites. We said, OK, for these four conditions, how comprehensive and accurate is some of the information provided on these health Web sites?

Our approach to this part of the study was really, if you will, to design a fair test to assess places of information on these Web sites. What we did was first for each condition, we convened panels of patient advocates and clinical experts for each of the conditions. We said, “Your task is to come up with 5-7 sort of 'need to know' if you will, topics and consumer-oriented questions.” So what we really said to them, if you were a content provider on a Web site, and you have a patient who has a question about breast cancer, what are the basic five questions that you think should be there?

Let's say if you have a car, the engine should be there. Then we said, for a question, we also want to know, what are answers or what are concepts that if a consumer were to go to a Web site with a question, what should be there? Sort of, what their bar-types of information should be there. This was part of the test. Let me just give you an example of one of the questions that the breast cancer expert panel came up with. The question was, “I have a lump in my breast. What should be done to check it?” Very basic.

The expert panel felt that these four elements below are types of information that a consumer should find on a Web site. Breast lumps should be brought to the attention of a physician. Mammography and ultrasound are useful in evaluating lumps. A negative mammogram doesn't eliminate the need for further evaluation. A persistent, non-fluid filled breast mass felt by a physician should be biopsied. These are the types of questions that each of the expert panels designed for each of the questions as well as sort of the basic concept that this should all be there.

After we designed these questions for each condition, we then had two searchers go to each of the Web sites we were studying and spend 90 minutes with the questions looking to see if they could find the answers. So if you went to a Web site and you searched for this information, the products of this research were then saved and put into a notebook. All information identifying the site was (unclear).

We then developed a standardized rating form, a scoring system if you will, and recruited expert reviewers to evaluate the information on two domains: coverage and accuracy. Was the information present at all and if information was present, what was the level of accuracy? This was a test that we wanted to make it fair and objective. Rather than just say, “Well, do you like what you find or do you don't like what you find?” Let me just give you an example of how we ask the physicians and reviewers to rate coverage for breast cancer screening.

The question that was saw earlier, “I have a lump in my breast.” If a Web site didn't mention mammography at all, it received a score of no coverage. But if a Web site said that mammography was a way to identify early-stage breast cancers, but it didn't tell you when the screening should start or if there was a difference in screening for high-risk populations, it got a score of minimal coverage. But if it mentioned that women over the age of fifty should have the early mammogram and talked about some of the pros and cons of mammography screening of different populations, it got a score of more than minimal coverage.

So these are the bars that we set for the reviewers to look at the information.

Cindy DiBiasi: So what were the results in this part of the study?

Dr. Gretchen Berland: OK, so what did we find? On English language Web sites, you can see what this shows here that this is the percentage of topics that the reviewers felt were handled really well. This meant that breast cancer did the best. Overall, two thirds of the topics that expert panelists felt should be on a Web site were covered well. Unfortunately, childhood asthma and obesity did not do as well at all. Barely a third of the information was covered well and on depression about half of that information was covered well on some of the more popular Web sites.

This looks pretty good compared to what we found on Spanish sites. In this instance, you can see that really almost the best condition was breast cancer, which was rarely covered as well as the worst topic for English language sites. What we don't show here is that nearly half of the topics that the expert panel felt should be covered on a Web site about these topics weren't covered at all. Nearly two thirds of the elements that related to depression weren't found.

Cindy DiBiasi: It is really fascinating. This is so profound.

Dr. Gretchen Berland: There is not only a difference in the quality of the information, but there was a difference in the quantity of information. For instance, if you were to go to a Web site and spend an hour and a half searching for information on breast cancer, on an English language site, on average, you will pull about 250 pages of material off that search. If you spend an hour and a half on a Spanish language Web site, you pull about 70 pages off.

There is not only a difference in quality, but there is also a significant difference in quality in terms of, you really point out an important point, there is a huge disparity in the quality of health information.

Cindy DiBiasi: Let's talk about the final part of your study.

Dr. Gretchen Berland: Right. This is the really important point, which is that let's say you can find the information. Let's say the information is there, but the third part of our study really asks the question, “Well, is it really going to be of any use to the population?” This is important because we know that the few Internet survey groups and the Harris Interactive Group reports that the Internet is becoming more and more representative of the U.S. population. That more people who are high school educated are going online. More Hispanics are going online.

What we did was we used widely accepted readability formulas which measure grade levels as a function of basically word complexity in a sample of text and applied these formulas to randomly selected passages of text in the Web sites that we studied. This is important because most people in the United States have graduated with a high school level education, but the actual average reading level in the U.S. is about the eighth grade level.

We found that on average, health information is not necessarily going to be accessible to many people. The average reading level on English language Web sites is the freshman level in college. The lowest level reading level on the Web sites that we studied was grade ten, which meant that for anybody who has a high school level education, much of the information they can find may not be useful to you because you can't read it. Spanish language sites did a little bit better. If you are a sophomore in high school, there were two sites that were acceptable that had a lower reading level.

Cindy DiBiasi: So what does this mean? What issues does this raise for public policymakers?

Dr. Gretchen Berland: I think this is the first study that really takes a slice of the technology that is very popular. I think it means that the promise of the Internet, many of us talk about it as serving as sort of the leveler or the Great Digital Divide and the ultimate tool that can really improve access to care for people. I think what we found initially is that we probably still have a little bit of ways to go.

Overall, we didn't expect to find the huge deficiency in Spanish language material. We didn't realize there would be such a variation in quality. We were surprised to find that there is such a variation by condition. Breast cancer, the good news is that for breast cancer overall the information is pretty good. We were a little disappointed to see that childhood asthma and obesity didn't do quite so well. I think also disappointing to us as well as the California HealthCare Foundation is that the reading levels on many of the health-related Web sites are probably still too high if we really talk about this tool serving as a tool that can really reach a wider, a broader public.

Cindy DiBiasi: It really needs to be taken down to the level of the consumer press almost, a typical newspaper reader might be 13 years old...

Dr. Gretchen Berland: I mean, the New York Times I think is written at grade 13. I think that the Web is such a fluid pool, that what we found is we are not developing the right content. We are developing contents for providing health information, or providing health information that is successful to people is really hard. But the Web is a place where you can provide information to a variety of levels, to people of different levels of education. I also mean this is also technology where the information doesn't necessarily have to be provided in text. Put it in video. You can put it in audio.

I think probably the study was conducted and we finished the study in November of last year. Since then, I know the Web sites have modified their sites. They have put up more videos. What we have found is that we set a bar and it probably needs more work.

Cindy DiBiasi: Great. We will be getting back to you because I know there are going to be a lot of questions on that. In a moment, we are going to open up the discussion for your questions. Please recall that you can communicate your questions to us in the following ways: if you are on the phone, just dial “14” and you will get right through to us. Or you may fax your questions to (301) 594-0380 or you may E-mail us your question at info@ahrq.gov. You may also directly type your question in the messaging field and hit “enter.”

Please also remember that if you prefer not to use your name when you communicate with us, that is fine, but we would like to know what State you are from and the name of your department or organization. So please indicate that regardless of the way in which you transmit your question.

Before going to questions, however, I would like to say a few words about AHRQ and the User Liaison Program. The mission of AHRQ is to develop and disseminate research-based information that will help clinicians and other healthcare stakeholders make decisions to improve healthcare quality and promote efficiency in the way that healthcare is delivered.

The User Liaison Program [ULP] serves as a bridge between researchers and State and local policymakers. We not only take research information to policymakers so they are better informed, we take the policymakers' questions back to researchers so they are aware of the priorities. Hundreds of State and local officials participate in ULP workshops every year. As a relatively new addition to the ULP portfolio of products, we hope that today's Web-assisted teleconference and the other two events in this Web-assisted teleconference series provided a forum for discussion between our audience of policymakers and researchers like those joining me for our discussion today.

We would appreciate any feedback you have on these teleconferences. Please E-mail your comments to the AHRQ User Liaison Program at info@ahrq.gov. Now we are going to go to our questions from our participants that are coming in.

The first one is to Pete. Is the public able to research a specific doctor's records through the South Carolina System?

Pete Bailey: No. We do not believe that the data systems are really ready for that in terms of the quality that exists there. I don't know when that world will be or if we ever want that world to be, but this is strictly for the provider. I think this question will continue to (end of tape)...to make sure that we have quality providers, but I think that a good part of healthcare is making sure that we don't destroy the bond of trust that is between patients and their doctors. I personally do not think that the average patient, I mean, I think we just had a great discussion about the difficulty of understanding.

I do not think the average person would be able to have the statistical ability to understand a lot of the issues and to be able to (unclear) like just for AIDS. It is just a very complicated way. I hope that our world doesn't go in that direction.

Cindy DiBiasi: One of the complexities is, aren't physicians afraid of being compared to other physicians who treat less complicated patients?

Pete Bailey: I had heart surgery several years ago and the night before I began thinking if I had scores on my physician and this guy was a B- or a B+, what would that do to me? I just don't want that world. I want us to have a world where we make sure that these physicians have the best quality information and that there are people monitoring that to make sure we have that quality.

Cindy DiBiasi: Gretchen, the question you said you get most about your study, what are the best search engines to locate medical information?

Dr. Gretchen Berland: I think that there were about 12 people who participated in this study who were listed in the JAMA [Journal of the American Medical Association] paper and probably every one of those twelve people has a favorite search engine and a favorite Web site. Just to answer your question, in our study there were really only two Web sites that statistically performed better on average.

One was for breast cancer, which was Oncolink.com, which was a site that is sponsored by the University of Pennsylvania. The other was the National Institutes of Mental Health site, NIMH.gov. No other sites that we studied statistically performed better. Those are sites that the reviewers felt covered; what it really meant that if you were to send 100 people to that Web site, that 95 percent of the time you would know that those topics were covered well.

Cindy DiBiasi: However, if you are going to NIMH.gov, you have got to be talking about mental health. You have got to be searching for those things.

Dr. Gretchen Berland: Those were the only two sites that we found in our study that performed better. The rest performed average and we couldn't statistically tell if they were any better or any worse.

In terms of search engines, there were two search engines that statistically took you to content that seems to be more relevant and one was Google and the other was Northern Lights.

Cindy DiBiasi: Karen from the V.A. in Michigan has a question about the study being published. We should have mentioned that the study has been published in the Journal of the American Medical Association.

Dr. Gretchen Berland: It has been published. It has been published in JAMA in the May 23rd issue. We have had a lot of reprint requests for it, but you can also download it if you go to the JAMA Web site, http://www.jama.com. You can find the paper there as well as a number of additional tables that were posted on the Web only.

Cindy DiBiasi: Anne, a question for you. Is there a way to query a private/public insurance payment by diagnosis?

Dr. Anne Elixhauser: Yes, indeed. That is one of the most basic searches that you can do in HCUPnet. You can look at diagnoses in a number of ways. You can look at them by individual ICD codes [International Classification of Disease], by groups of ICD codes, that you may find there is a certain series of ICD codes that describe the condition that you are looking at.

You can look at diagnoses by what we call the Clinical Classification Software, which is an aggregation of ICD codes that we developed here at AHRQ. Also by DRG [Diagnostic Related Group]. Then you can look at the patient's insurance status, whether they are Medicare, Medicaid, commercial, uninsured. It classifies patients that way. That is indeed one of the most common searches that can be done using HCUPnet.

Cindy DiBiasi: Do find that there are the top three ways that HCUPnet is really being used? Is there one thing that really stands out?

Dr. Anne Elixhauser: I think probably the most common use of HCUPnet is to look at a very specific condition and then to get any kind of detailed information that HCUPnet provides. For example, you can look at a condition like depression or septicemia and you can build a fairly comprehensive profile of what is happening in hospital care for that condition. For depression, for example, you can look at length of stay, the charges for hospital stay with depression, how many patients are hospitalized for depression.

Then you can break it down by all kinds of patient characteristics, comparing males and females, comparing different age groups. Then you can look at it by peer as a previous question just illustrated. You can look at it by low-income vs. not low-income patients. You can compare types of hospitals and what they do in terms of the treatment of depression. You can also look at trends in depression over time and you can compare national, regional and State, all those estimates, for depression on the national, regional and State level as well.

Cindy DiBiasi: Pete, I know you are also working on a plan to link data from a number of your State's existing health-related databases so that this information can be better used to support policy decision-making. Can you talk a little bit about that?

Pete Bailey: Yes. This is really an exciting area that we are involved with. Our office is, as I mentioned earlier, a research and statistics agency and as a neutral agency, we have been able to pull together an enormous amount of data from different agencies. What we have found is that, our belief in general is that a high percentage of the people with problems in society that we are attempting to try to solve with our programs have multiple problems that cut across specialty areas or lines of authority or how we can use our resources.

Those problems might be health or poverty or education or training or inabilities to get a job or alcohol and drug abuse or mental illness or family structure or child support or law enforcement. If you look at the slide that we have up now, all of these agencies including the private sector system, which you see down at the bottom right hand corner. We have their data in our computer down to the individual level and we have developed a unique tracking number where we can track people across these systems. It enables us to link health and social services and poverty and a whole range of issues.

What we are doing with this, of course having this is one thing and being able to do special research studies off of it is another thing, but what we are developing is a Web-enabled data warehouse where users must have a certificate to enter the system and that certificate would control what they have access to. What our desires would be for any of these agencies to be able to come into the system and be able to take a cohort and query and see that, be able to ask as many questions and cut across systems as possible.

In the next slide, what we are trying to show is the way we are thinking about this, and that is, how can people access all of this data? One of our biggest challenges has been the enormous amount of data. How can you set up a system where people can come into it and get that kind of information? We have set it up where they can either come in through a State agency program or service. They can come in through special populations that we have created, like children with special healthcare needs, either through geographic profiles or they may be interested in a particular zip code or a block or block group.

We are also setting this system up so that we would be able to do analysis at the House and Senate level which will be really interesting to be able to do that. They could come in through some demographic group like “Children under 6” or through healthcare utilization like you are interested in, only diabetes and you want to look at those patients. It allows people to track through this kind of information, which I think has tremendous advantage.

Cindy DiBiasi: Gretchen, let's just talk about some of the research steps that you are now looking at.

Dr. Gretchen Berland: Let's go to slide eight. I think it is important to point out that probably many people online, I suspect that, I gave this recent talk and asked 500 people who went to the Internet to look for health information and nearly every hand came up in the room. I think our first step really, we just classified what you might find out there in a certain place or time. I think, as this slide will show, that we really don't understand yet how health information influences the doctor/patient relationship.

There is a lot of anecdotal information about patients bringing in more health information in to their doctors from the Web that we haven't really studied how it influences utilization. Whether patients are asking for different things, if they are asking for inappropriate things or more appropriate things. We haven't studied that. Probably most importantly, we haven't studied how health information is influencing healthcare outcome. Is it improving people's health? Are people taking better care of themselves? Are the decisions that are occurring as a result of health information having a better impact on their health outcome or a worse impact?

Cindy DiBiasi: We certainly know anecdotally that patients are walking in with reams of printouts from those to their doctors.

Dr. Gretchen Berland: Right. They are. I have a number of colleagues who are oncologists where, even when I was training in medical school, part of the standard medical history was, there is a whole kind of series of things you need to ask. Now what a standard question is “Are you going to the Internet to look for health information? If you are going, to what Web sites are you going?” That's something that you... often people will come in with chemotherapy regimens that they are asking for because they found it on the Web. This is new.

Cindy DiBiasi: Do you have recommendations? Is that part of the study or a followup to the study? Recommendations on how consumers can use these Web sites more effectively?

Dr. Gretchen Berland: That is an important question. I think it also points out one of our study limitations. Partly in research, you would like your study to be able to answer all research questions and this was just a first step. We hope that there are plenty more. Our study did not use consumers. We sort of, we basically broke it down in a very structured way. What you might do on the Internet and we tried to mimic what consumers might do.

I think there needs to be more studies that look at how consumers use health information. How consumers use the Internet. I know the National Cancer Institute has a lot of usability testing studies where they, how do you place things in a way that it is easier for people to use? That may mean maybe for the elderly or people who may potentially have limited vision. So how consumers use health information is really some of the next steps that either we will be doing or we hope that other groups are doing.

Cindy DiBiasi: Wow. Anne, a question from Jean. She wants to know what States are participating in HCUP.

Dr. Anne Elixhauser: I was afraid that question would come up and I of course don't have a list. I am sure this is like the Academy Awards and I am going to forget somebody. Like I said before, we have got 24 participating States right now. States from all corners of the country: Arizona, California, Colorado, Connecticut, Florida, Georgia, New Jersey, New York, South Carolina, Washington, Wisconsin, Oregon, there is probably more.

Cindy DiBiasi: You did them almost entirely in alphabetical order. That's impressive. (laughing)

Dr. Anne Elixhauser: At this point, we have about 90 percent of all hospital discharges from the Northeast and from the West. We have got really good coverage there. We have got a little less than 50 percent of discharges from the South and the Midwest. We are trying to really build up our participation in those two areas.

Cindy DiBiasi: What future enhancements are you planning for HCUPnet?

Dr. Anne Elixhauser: Probably one of the biggest things that we are going to be adding in the next six months or so is quality indicators. Pete mentioned those a little bit earlier. A number of States have been using these. What we did a couple of years ago is a group of us here at AHRQ, in response as a matter of fact to the HCUP [Healthcare Cost and Utilization Project] partners, developed what we referred to as the HCUP Quality Indicators. These are indicators of healthcare quality that you can look at using just hospital administrative data.

We have recently just released a study that Stanford University has completed for us where they expanded and enhanced what we referred to as the next generation of HCUP Quality Indicators. What Stanford has done is come up with 45 measures. Things like in-hospital mortality for specific conditions or procedures. Volume of procedures for those conditions where there has been a demonstrated link between the number of procedures that are done in the hospital and the outcome of care, such as in-hospital mortality.

We have some measures for utilization of procedures for procedures where there is some question about the appropriateness of care, like C-sections and hysterectomies. There is also a group of indicators looking at what are referred to as ambulatory care-sensitive conditions, which are indicators that don't even look at what is the quality of care inside the hospital, but are using hospital data to get at potential primary care problems. Problems in the delivery of care in the community. For example, pediatric asthma is a great example.

Most kids should be able to be treated on an outpatient basis so that they never have to be hospitalized for their asthma. If there is a high rate of asthma hospitalizations for children, there is something going wrong in the primary care those kids are receiving. Other measures like that would be diabetes complications. Again, people with diabetes should be able to be treated outpatient without getting admitted for short-term diabetes complications like coma and ketoacidosis.

Stanford is continuing development. They are going to be releasing these later in the year, what we refer to as Patient Safety Indicators or Complications of Care that can be looked at using again hospital administrative data. We are going to be applying these quality indicators to our HCUP data. That will then be put on the Internet so that individual States can have national and regional comparisons. How does my State compare on these indicators with the nation as a whole or with other States in my region? Or individual hospitals can compare their performance with specific peer groups. Other teaching hospitals in the Northeast, for example. That sort of thing.

Cindy DiBiasi: Let's talk a little bit about the HIPAA [Health Insurance Portability and Accountability Act] requirements. Pete, let's start with you. What are the implications of those requirements on your initiatives?

Pete Bailey: We are certainly scared of what they might do to us and I certainly am no expert in that area. The way we are doing our best, we have done this before with HIPAA. We developed a unique tracking number. We used identifying information to develop this tracking number, but when data come to us, we separate the identifying data and the statistical data. We use the identifying data to develop the tracking number and then we link that tracking number over to the statistical data set.

If we get data on you from four or five different agencies or sources, you would have that same tracking number. We never put any information together about you on anything that would identify you. Not in our office, not anywhere, because I think that does break the law. The other thing that we think we are in pretty good shape on is that the work we are doing, we have laws that require the data and that also I think (unclear).

Cindy DiBiasi: Anne?

Dr. Anne Elixhauser: I guess HIPAA affects us in a couple of ways. One is the standardization of healthcare information. That actually is going to help HCUP a lot because we spend an awful lot of resources right now just standardizing the data that we receive from the States, converting it into a uniform format so that we and other researchers can do analysis with it. Once HIPAA standards are in place, we can basically tend to other, more interesting things. Instead of just standardizing the data, we can do more research and spend our resources in other ways.

With respect to privacy, there are a couple of ways that HIPAA affects us and a couple of ways that we have sort of dealt with it already. HCUPnet, for example, does not use, it is not built on the data itself. We don't have patient-level data that underlies HCUPnet. HCUPnet is based on aggregate statistics. It is based on principles of data warehousing where we have these huge data cubes that we have basically compiled all possible combinations of data and made that available through HCUPnet.

One of the reasons why HCUPnet works so quickly, if you had to actually do the principal analysis right on the data itself, it could take 45 seconds to a minute to get your results, which would be deadly on the Internet. It just wouldn't hold on to anybody. Because we have these data cubes and because everything is aggregated already, whenever you do a search, most of the searches come up pretty quickly.

The main reason that we did that was because we didn't want to have the patient-level data on the Internet so that a hacker could get in and actually get access to the discharge-level data. The side benefit of it was the speed that it provided. The HIPAA privacy issues are affecting us just in terms of I think we're mostly just concerned about the extent to which data can be shared with us. There are a number of people who are on the HCUP staff who are working very actively with a number of the States who are concerned about this, trying to understand the HIPAA regulations.

There are a couple of people on our staff who have actually helped to draft some guidances that hopefully will be released as part of explanations for the HIPAA regulations so that it explains exactly what HIPAA means in terms of sharing data with us and then us making data available to researchers. 

Return to Contents
Proceed to Next Section