Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

2012 Meeting of the Subcommittee on Quality Measures for Children's Healthcare

Transcript (continued)

Transcript of a meeting of the SNAC held September 12, 2012, in Bethesda, MD.

Agenda Item: Scoring and Voting

Charles Irwin: We want to establish what the rules of the road will be for the rest of the meeting. I think we have probably beat you all over the head with this several times, but just to reiterate some of the key points to first of all this improving core set is not a new set. The core measures recommended today will be recommended for consideration to be added to the initial core set. There are other recommended measures that we will also recommend for consideration for other public and private programs. We are not bound by having to complete our work today or even in the next year. We have 2 years. It was a 3-year sentence that everyone got. Just remember that when you say yes to anything.

The measures that are not recommended today can be reconsidered, as we have talked about, for 2013 or 2014, and the nominators will have the ability to resubmit. As people talked about these domains and whether we need to make certain that everything is covered, that is not necessarily our whole job today. I think it is our job today to look at the current measures that are under consideration.

The other thing that I wanted to just reiterate is some of us do have conflicts of interest in terms of voting. When we come to the measures if you have a conflict, if you have been engaged in the development of that measure, if you are a close collaborator with those individuals, we hope that you will refrain from voting. It is kind of the same process that many of us are engaged in when we review grants. Or if they come from our institution, and the individual's office is next door to us, and they run and show us their measure before they submit it, there is a conflict.

We have already done the introductions. We are going to turn to the key business of scoring and voting. This is what you all have been waiting for. And I want to try to explain this, and Denise is here to help with this process.

Our first step at least today is to really go through a Delphi III process in which we are going to be looking at measures that scored 7-plus but also have this high degree of dispersion. And there are 10 of those measures that we are going to look at and then vote on whether we want to bring those forward for step two, where we will consider all the measures that currently have made it through the process.

And then in the third step, we will think about what the cutoffs are for acceptance of a measure for the improved core set, and we will also vote on that.

And then we will actually kind of parcel this off to what we want to move forward for consideration, and what we want to put in the Medicaid set, and what we want to recommend for other public and private purposes.

In order to do this, we are going to start with this Delphi III process. We determined that the measures that past Delphi I and Delphi II had to score 7 with a low IQR [interquartile range]. It had to be below 2.5. Now, what we have is this group of measures that scored 7, but they have a high measure of variance. What we would like to do is look at each measure and then ask individuals to vote on these measures. But we would actually like to have some discussion around these measures because I think some of us will not remember exactly what the measure is. And we really encourage the group to bring their thoughts and ideas about considering these measures.

And then we will try to establish how we want to determine the cutoffs of this. We do not have the ability to do the 1 through 9 measurements here. Our recommendation is that we have a 1 through 3 scale. We think about the 1 through 3 scale in the same way that we thought about the 1 through 9 scale. Number 1 would be 1 to 3, kind of a low priority number. Then 4, 5, and 6 would be the kind of moderate medium, and then the 7, 8, and 9 would be number 3. Essentially, we are trying to get some kind of equity or kind of similarity to how we voted the last time. And then we can score these with our little whatever it is. We can pretend we are at the convention by voting for whomever we want to vote for. We would score that.

And then we have to decide actually, what is going to be—what are we going to expect or what are we going to demand from this. Are we going to demand that a measure scores 3-plus, or 70 percent of us vote for it to go forward, or are we going to be more liberal? We were pretty tough the last time. I think we ought to be consistent with the way we vote. I open that up for discussion right now because we have to make that decision before we can even go forward at this point.

Gerard Carrino: Since we are going out of the starting gate right now, we could be very conservative since we have 2 years, which is all the time in the world I believe and go with the 3-plus. I do have a question though. My little token has five pieces on it. Do only three of them work?

Denise Dougherty: Do we have a way to vote yes or no on the 70 percent cutoff so that people do not have to—everything else has been anonymous. Good practice. Exactly. Thank you.

Charles Irwin: Will the majority rule then? We will do majority rules.

Clint Koenig: Somewhat related, but will there be a process somewhere down the road where we can vote on retiring measures, or will there be a process by which measures are retired in favor of improved or improving measures?

Denise Dougherty: I think it is probably more in the future. For example, this time we have a couple of measures on C-section, but there is already one in the initial core set. We could decide whether the ones that have been proposed today are better than the ones—in that case, the one today would replace the one in the initial core set because you do not want to have States recording two or maybe you do because one is at the provider level and one is not. That kind of discussion.

Karen Llanos: I would also say there is one measure that we know is badly coded or hard for States to collect just based on the codes, and that is part of the initial 24, and that is the otitis media one. And it is one that we have actually put on hold in the reporting template. States are still able to submit the data, but we have already filed that issue. That would be potentially one where it would be wonderful to have an element-related measure and efficiency measure that could take its place.

Denise Dougherty: And one of the Centers of Excellence, I forget which one now, is working on enhancing that measure.

Glenn Flores: We had a little bit of a discussion about this in our last call, but there was a concern expressed about just arbitrarily taking numerical cutoffs because of a couple of issues. One of them is you might have the wrong measure for the right indicator. Some of us may have scored the quality of the measure in front of us, but it addresses a very important issue. It raises the other issue of should we do a sensitivity test and say we are going to an initial cut at 7, but let's look at what the 6 to 7 that we are going to leave out look like?

And I know we have talked about if measures are not up to snuff, they will be recirculated and go through the sensors. But I just want to make sure there is a clear process because I want to make sure I know what I am voting on. If I am voting on, I do not like this measure, don't ever want to see it again. That is different from this, it is an important area. We did not hit the target right now, so then what?

Charles Irwin: I think it is a really good question because there are some areas that I think are really fantastic, but I think the measurement is not there that we have been given. I struggled with that in terms of the scoring.

I think the issue is—I would argue that we have to vote on whether it is a really good measure. But I think that if the area does not seem that important—measuring something that is not important—I do not think that is what we should be doing.

If we go back to the 6 and 7 and start considering all of those measures that are there, I think we will never move the task forward. I think those measures that are in that category we will take a look at later this afternoon. I believe we ought to think about of those, which ones would we really like to perhaps encourage individuals to come back with or maybe we do not want to encourage. Maybe just leave that up to their own decisionmaking if they want to bring it back.

Francis Chesley: I think that is an excellent point, Charlie. I just wanted to mention. As you go through your voting, if you identify those measures—let's just say in either of those two buckets there may be others. We have folks around the table who will capture that, and then we can have that feedback with the measure submitters so that we do not lose those nuances.

Rajendu Srivastava: I am sorry. I may be a little lost, which might just be me. I just wanted to be clear. We are talking about 7-plus to vote on. And then we are going to have a separate time this afternoon to go over measures that did not make the 7-plus but might have had some very important implications. Although the way the measure was explained might have been lacking in certain areas. So by us being able to, from a SNAC point of view, provide almost like in a grant review where you could say you came close, but these are aspects that if you could go back and work on, might be more favorable next year. Is that what I am hearing?

Charles Irwin: Yes. The first voting for the Delphi III process, we still have these measures that scored 7, but had this high IQR that we did not bring forward. We have to make a decision with these 10 that still are outstanding, whether we are going to bring those forward for the final consideration. I think that is really important.

Then after we do that and after we make the decision regarding all of the ones that have come forward, we can go back and pick up those other ones for consideration and discussion.

Denise Dougherty: The ones for Delphi III are in the handouts that you received. You will see them.

Charles Irwin: Let's just vote on this measure first, and then we will go. Can you tell us how we do this now?

Staff: Everyone should have received a key fob who will be participating in the voting, 1 through 5, and you will see it is 1 through 5 or A through E. For right now, for the purposes of the voting, we will be using 1 through 2. It is going to give a 10-second countdown, and you can vote any selection. You can change your vote, but it is going to record the last number that you choose. You will press on the button and it will be a little green light. To ensure that your vote has been cast it looks for that little green light indicator. Just a reminder, you can press 1, you can press 2 within the 10 seconds; whatever is the last option you chose is what will be recorded. Does anyone have any questions?

Maushami Desoto: I think we will do a test run just to see how it works.

Participant: Should we do it now?

Participant: When I click on here, you will see the countdown begin.

Charles Irwin: We need to vote.

Participant: So 3 is the equivalent of 7 to 9 in the Delphi 1 to 9 scoring.

Charles Irwin: Alright. We have made the decision. It is going to be 3 or better. We will move on.

I think each measure is open for discussion right now. We do not want to just vote automatically. I open this up for comments regarding this measure.

Denise Dougherty: I do not think we have the numerator and denominator in the notebook. The numerator statement is for Medicaid and CHIP standalone programs. The numerator is the summation of covered days for all children over an 18-month observation window calculated for Medicaid and CHIP separately and does not reflect transitions between programs. But this measure may also be calculated as program-specific. Numerator exclusions. Children must be age 0 to 18 at the beginning of the 18-month observation window.

Denominator statement: The denominator is the summation of eligible days over an 18-month observation window for all children appearing in Medicaid records at any point within the 18-month observation window or an 18-month look back period calculated for CHIP separately. There may also be cases where MCHIP and SCHIP data in the State may reside separately. It is a little bit more than the denominator. Does that help, Diane?

Diane Rowley: Do States have the ability to look at number of eligible of days in 18 months to actually come up with a denominator?

Kim Elliott: In Arizona, we do know who is and who is not in Medicaid and when they are on and when they are not. But my question on this is, what are we going to do with that information? It is good to know, and it would tell us why we are seeing the results we are, but it is not necessarily an actionable item from a State perspective, as you would not necessarily have to change your enrollment and eligibility criteria. It is good for policy, but I am not sure that it necessarily would impact the outcomes.

Stephen Saunders: Actually, I would disagree. I think this is a measure that would help States understand their enrollment process and how easy it is for families to actually enroll and stay enrolled in this measure, and then the duration one that follows it, I think are companion measures. Is the eligibility process easy? Is the re-enrollment process easy? Is it not easy? Is it automatic, or do they have to resubmit the paperwork and all that kind of thing? In the end, if you are enrolled and covered then of course, essentially you should have better health because you have access to medical care. I think they are both quality measures but also State process measures. This is a combined measure, I would say.

Denise Dougherty: This is a specific request to have a measure in the CHIPRA legislation. It is not your typical clinical quality measure, patient experience of care. It is a specific request within the CHIPRA legislation to have a measure like this.

Glenn Flores: Can you qualify which measure we are talking about?

Charles Irwin: It is measure 78 and measure 79. We are looking at 78 right now.

Glenn Flores: Which one is that because a bunch of us did not know which measure we were talking about?

Charles Gallia: One of the questions is, can States calculate member days? I know that our data system structured it so we would know the duration of eligibility based on spans of time, begin dates and end dates. It is possible to do that.

We do have duration of coverage measures that are in EPSDT [Early and Periodic Screening, Disgnosis, and Treatment]. And also, I was wondering about—it says 18 months too. There is some decisionmaking that I was not exactly sure about why there is an 18-month look-back period instead of an annual one. There are some disconnects from some of the other things. I just did not understand the rationale behind that particular decision. When I use duration of enrollment, I go back to what the period of eligibility is that is in EPSDT as a standard. It would be something that is in placement. There is a duration of enrollment or periods of enrollment that is an EPSDT report that States submit. But it is not part of the CHIP measures.

Andrea Benin: Are you just saying that States already report this measure? Is that what you are saying?

Charles Gallia: Not this measure. What I am saying is that States have a process for calculating coverage duration, and it is part of one of the things that they submit in their EPSDT reports. It is just knowing that what we are going to have with this, there would be another one that would be out there that is used as a numerator and denominator calculation and a whole different reporting structure that we did not even mention.

Andrea Benin: There is another measure that gets at something similar that people are already using. I am not understanding.

Denise Dougherty: Is it standardized across States?

Charles Gallia: I would say it is as standardized as this. The program eligibility—if a person moves between programs, this measure—in the CHIP program, the eligibility—CHIP has such a narrow band that people move in and out of that program a lot. But when they go into Medicaid, they stay there because there is a larger income distribution. The movement between those programs is fairly high. The way that this is structured is it has it isolated. It is CHIP and Medicaid. The duration of enrollment that is going to be—you are going to see a lot of movement in the CHIP program and not so much in the Medicaid program, and does that really tell you what you need to know and that is the overall care continuity that is the underlining question and the purpose of the measure.

Stephen Downs: I think that comment pushes us into the aspirational side of trying to get this done, which is a measure like this—although it is going to have the shortfall of being overly sensitive to kids who are churning in and out of one program or another one—it will have the more aspirational goal of encouraging longer periods of contiguous enrollment, which has been shown to have some improvements in ongoing chronic care and preventive care management.

Charles Irwin: Other thoughts about it?

Francis Chesley: I am going to ask Jeff to clarify a point about this issue of Medicaid or CHIP.

Charles Irwin: I just want to define a process issue here now, Francis. We have the Centers of Excellence here. I really value their input. But I am concerned that—are we violating any rules by allowing people to speak to their measure if they are in the room, although this is a public meeting? In public meetings, people should be able to speak. I just want to be clear on what we are doing and be consistent with the measures that we are discussing.

Francis Chesley: That is an excellent question. Here is how we have thought about it. As you said, this is a public meeting and also in the Federal Register announcing this meeting, measure submitters were invited to participate in the meeting. I think we have to be very careful though about the folks who happen to be in the room as to what guidance they are allowed to provide for the SNAC in your deliberations. My conception of how we walk this tightrope is to say that if there is a clarifying point that can be helpful for the committee as it deliberates on a measure, not so much defending the measure, but answering a specific question. For example, are we making appropriate assumptions about this measure and its attempt to capture transitions between Medicaid and CHIP? If there is a way to answer that question, yes, that is what we are doing or no, that is not what the measure is doing, without defending the measure and its importance, I think that is something reasonable that we can do.

At the same time, we are going to get back to measure submitters similarly who are in the room for whom we have questions about measures that we think are important.

Charles Irwin: I would like to establish that I do not think the co-chairs should be—it should either come from the members of the SNAC committee or people in the room to come forward. But I am not going to go to people that I know in the room that maybe submitted these measures and ask for them to come forward.

Francis Chesley: Your preference would not to have clarifying—

Charles Irwin: No. My preference is a clarification, but it should not come from the co-chairs. I think that it would be fine.

Francis Chesley: In this case, Denise happens to know. I think it seems like a stumbling point, but I think you are right. I think we have thought a lot about this and have tried to figure out how we can titrate information for the committee in fairness and objectivity. If there is a specific question.

Charles Gallia: I am left with two questions. One is the period of time and then the 18 months and the decisionmaking regarding that versus an annual measure. And then the other is the churning between programs and the sensitivity and having reports that would be isolated which essentially would be constructing two sub-measures, not just one. If the steward of the measures is available to respond to those questions, I would appreciate the response.

Participant: Thank you for giving me the opportunity to respond. First, in terms of the churning, the conceptualization and the measure were about insurance. We had thought of it as a combined CHIP or Medicaid enrollment. It is really not an issue of churning. It is an issue of during a certain period of analysis, in this case, it was 18 months, how many days were you insured in some way be it CHIP or Medicaid? I am not sure if that answers the churning question, but I think that clarifies it. We calculated these metrics based on the data we got from MAX, and some of the MAX data allow you to get that information.

In terms of the 18 months, we debated between 12 and 18 months. We thought 18 months was best because of the look-back period that would allow us to detect anyone who had insurance at some point. We thought 1 year was not quite long enough to really make sure that the concept of coverage got at the point that if someone was ever eligible, we would like to know if they continued to be covered. We did look at the 12 versus 18 months. We thought 18 months was a better metric. Since all of these data are calculated off MAX data, generally those data are available and it is trivial for the—I should not use the word trivial. But it is not hard for programmers to change the window from 12 to 18. We thought about 12 versus 18 and thought 18 was a better number to use. We have talked about that somewhat in the narrative.

Charles Irwin: Thanks a lot. Any other further comments regarding this measure?

Denise Dougherty: Not that I want people to go into their computers right now. The comments made by SNAC members while they were doing the review are on the WebEx.

Charles Irwin: It is also in the notes that were handed out to us. On Table 4, the comments are here. I would like to have someone nominate that we bring this measure forward for voting and then second it and then we will vote.

Stephen Saunders: I will go ahead and nominate. I think we should move forward and consider this measure.

Denise Dougherty: Second.

Charles Irwin: I guess we should vote. Remember the range 1 to 3.

Andrea Benin: Somebody made a comment about how—maybe it does not matter for the purpose of this, but that this measure might not discriminate between economic upturn versus economic downturn. I just wonder if somebody could expand on that a little bit. Is that important to our thinking about this? I am not sure if there is anyone who can clarify that a little bit.

Charles Gallia: I know when we have been monitoring enrollment in our programs, that with the recession we saw an increase in the number of children that were entering CHIP as a result of it that had previous insurance through private or uninsured. And then over time, there was movement between CHIP and that actually goes into becoming eligible for Medicaid and closer and closer to having no income over a duration of time means that you are going to lower your household income and then qualifying moving it downward. In the opposite, when there is—prior to the recession, I saw an increase in people moving out of CHIP, losing eligibility because their incomes were increasing at higher rates. And they would move into something else. It could be private insurance or another status. Since CHIP is a narrower FPL band, you get a lot more movement in between, both down and up. In a recession, you would see more children moving out of CHIP over the long haul into Medicaid, and then in the reverse, we would see them moving out and losing coverage because of CHIP into private or other options. Does that answer your question?

Andrea Benin: It is a ratio that if you are eligible in month 1, you get into the denominator, and then if you move out in month whatever because you are doing better economically, you are still in the denominator. The ratio potentially goes down in an economic upturn.

But there is also a scenario where—how does it play out with patients who become uninsured? That is what I am not totally getting my mind around. If you have more uninsured patients, are you eligible? How do you get found to be eligible? How are you identified as eligible if you are in that uninsured group? Are you then missing altogether? Is this metric really telling you something about economic status and access to health insurance? I am a little confused about how this metric plays and what it would tell you.

Stephen Downs: Let me comment on that. I am not sure if you are looking for the developers to comment on it. The issue of loss of coverage despite eligibility really has more to do with the redetermination processes and the administrative steps in retaining your Medicaid or CHIP insurance from the redetermination period. For example, if a child becomes eligible for Medicaid and CHIP and that eligibility is considered continuous for 18 months, you would automatically meet these criteria, the denominator and the numerator.

If every child who came on to Medicaid or CHIP was automatically eligible to remain on that without redetermination for 18 months, it would be very difficult not to achieve extremely well on this measure. On the other hand, if redetermination is triggered every time a child enters or exits some public program and they enter into an arduous registration process to retain their eligibility, you will have people drop off. That actually has been demonstrated.

There are numerous ways that meeting these measures could change administrative processes in a Medicaid or CHIP program in order to improve this quality measure that really have to do with duration of time before redetermination of eligibility and ease of getting onto the Medicaid or CHIP roles. Does that make sense?

Glenn Flores: Could I make a comment about just the indicator in general? I think it is absolutely important. I think it is one of most important indicators we have here, not only because it is in the CHIPRA legislation, but because two-thirds of American children who are eligible for Medicaid and CHIP are not enrolled in it. I think this needs to be a quality indicator for systems, for health plans, and for States because otherwise, a lot of the other things we are doing here today will not necessarily make sense. I think this is really important. I felt it had to be said.

Carole Stipelman: I want to back that up. I come from a State with a very high rate of churning. And my State is not going to give those data up. I would love to get my hands on the data.

Charles Gallia: I cannot speak to the experience that you have in other States, but I know in our CHIP annual report, we have a method for assessing the rates of successful coverage and performance bonuses as a result of our outreach. The number of uninsured children in our State and the ones that are potentially eligible and the coverage that is provided, those are formulas that essentially are calculated, that we are assessed whether or not the State qualifies for its outreach efforts. I am not sure where the two-thirds came from because in our State, we have 98 percent coverage of the CHIP eligible population. Nationally, I do not think it is one-third of the children eligible.

Glenn Flores: Actually, there is pretty solid evidence from a Health Affairs article a few years back. I am happy to share that with you. The States do vary. For example, in our State of Texas, which is number one in uninsured children (laughter)

Participant: Forty-nine States and then there is Oregon.

Glenn Flores: It is pretty solid evidence.

Charles Irwin: This has already been nominated to move forward. Are we willing to now vote? Let's vote.

(Vote taken)

Charles Irwin: The comments regarding this measure are on Table 4 page 2 or else on your SNAC console if you have your computer here. This looks very much like the last one. It will give you the percent that are continuously enrolled at 6, 12, and 18 months, which I think would also be another valuable indicator of State.

This is once again looking at State health or State policies regarding enrollment and redetermination and how easy or difficult it is to stay on the program. Like the first measure we just discussed, this would be a similar type of measure that is really going to be looking at State processes and how patient-friendly they are in terms of staying on the program, how easy it is to stay in the program once you get on.

Glenn Flores: Another comment, I am sorry if I keep chiming in about insurance, but it is something I do a lot of work on. There is really good evidence about episodic insurance coverage and the fact that there are twice as many kids who are episodically uninsured versus just uninsured the full year. And we know from a New England Journal article from last year that their access to care is as bad in many instances as those who are continuously uninsured. An extremely important measure, I think. I am coming from a State remind you in Texas where we require everybody to renew their Medicaid every 6 months. I think for those States, it will be interesting. You will probably see variation versus States that have a 1-year renewal, but I still think we need to hold the States up to that quality standard of keeping it continuous.

Naihua Duan: I guess there are overlaps around the measures. Are we coming back later to discuss them?

Charles Irwin: We will come back to discussing the overlap when we get to moving the measures forward. My recommendation now would be not to be so worried about overlap because there are even measures that have made it through our Delphi I and II that are really overlapping measures. I think at this point we should really be voting whether we bring these forward to the final vote. That would be my recommendation. It is open for discussion.

Stephen Downs: Can I ask for some clarification? Maybe I need this from the nominator. But there are references to enrollment in January of 2006 to December 2006. Maybe I am missing something. I am not sure what the reference to 2006 is.

Participant: Thanks for the opportunity to clarify, we are the nominator for the developer for this measure. The data that we presented are based on the MAX data that we obtained. That is why you have the dates and times there as an example.

Stephen Saunders: Are you looking for a nomination on this measure?

Charles Irwin: Unless there is any further discussion.

Charles Gallia: It is just an observation about the observed first enrollment. It is a definition in terms. But I think if we address this, the definition of a new enrollee first observed with the look-back period of 30 days was a little bit problematic for me because I know that most—HEDIS [Healthcare Effectiveness Data and Information Set] measures, for example, use 45-day break instead of 30. It is a nuisance.

Participant: I am happy to clarify. We actually used the rubric that was developed by CMS that specified 30 days for a look-back period. It certainly could be changed to make it 45, and that would produce slightly different results, though I am not sure they would be radically different.

Charles Gallia: In some of the measure specifications, the reason for the difference is because each State has a different review cycle, whether it is weekly or monthly. If they lost coverage for a day, it could be that the reenrollment is for a month out. That is why they go to the 45 days to include those that would be just a 30 day because they missed 1 day. It would be a loss of eligibility that would include administrative changes as well.

Participant: Absolutely. We understand.

Charles Irwin: Steve, you recommend that we bring it forward?

Stephen Saunders: Yes.

Charles Irwin: We need someone else to second.

Gerard Carrino: Second.

Charles Irwin: Okay.

(Vote taken)

Charles Irwin: Based upon our decisionmaking tree, we will not bring this measure forward.

Next measure. Again, page 3 of your handout, Table 4 lists some of the comments from the SNAC committee members.

Diane Rowley: When I look at this measure, I take into consideration the fact that there are a lot of well-baby visits in the first year, and that the health of the infant during the first year is very important. I actually do not see this as overlapping with the other question about coverage because it has a high importance and particularly since we are looking to see how care affects infant mortality.

Carole Stipelman: I would like to add that the most vulnerable time to drop off of Medicaid is at 1 year of life. Babies tend to get enrolled in the hospital when their moms deliver, and then re-enrolling at 12 months is a fragile process.

Feliciano Yu: I have a question maybe for the nominators. Again, this measure is derived from the first measure 78 if one has the data.

Francis Chesley: Before you get an answer, I just want to make a process point. Recognizing that we have some measure developers who have proposed measures in the room, and some who do not, I think it would be more clean if we did not direct questions to the measure developer but instead have a discussion. And if that discussion generates a clarifying question then we can pose a question to the developer.

Feliciano Yu: Point well taken. To the group then, is this something that we need to consider whether this measure or some of its measures can be derived from existing ones that we have so that we are not redundant. We are looking at the same data point or periods. Is there something that can be derived out of that? That is more of a question.

Gerard Carrino: I think that is a wonderful question, and I support asking the question. I also want to echo what was said earlier in the discussion, which is this particular measure is so important because of the fragility of the first year of life that even if the answer to the question is no, even if it means that the burden is greater on States because it is not part of the same data set, I am planning to vote for it.

Kim Elliott: In response to the question that you had, I think that yes, we could definitely get it from 78. You know your age breakdowns and you know who is and is not covered. Yes, you could stratify it that way.

Stephen Downs: Having not actually tried this, my interpretation of these is that this is a strict subset of 78, which I think is your point. But I guess the argument here as I am hearing it is that it is such an important subset of the measurement in 78 that it may deserve its own set of markers.

Andrea Benin: I am reading these as really different. I am reading 78 as having the denominator of patients who are eligible and then a numerator of the ones who are at the time of enrollment, but that these other ones are actually not eligible, but patients who actually got enrolled and then at different time points were still enrolled. So, 79 and 80 seem to be if you were enrolled as the denominator, and 78 is are you eligible? Those are conceptually different. I am far from the expert. I am trying to make sure I am understanding this.

Charles Gallia: They are different. One is anchored to the duration of time in a program. This is anchored to an individual, even though you accumulate the time of individuals in the other measure. I think this is a great distinction from the previous one. A newborn, we can identify in claims data administratively. And we can look at the duration of enrollment for those newborns. Those are not administratively complicated in terms that the data do exist, and for a population as you have said, are pretty particularly sensitive. Unlike the other measure that was generic about what is really new or not, this one is fairly clear.

Charles Irwin: Shall we vote?

(Vote taken)

Charles Irwin: This measure will come forward.

Glenn Flores: Can I ask you a procedural question? On measure 2, it is going forward. I am wondering if when we have a measure that does not pass, can we now put it into a parking lot for talking about it later if the group feels we should or if we make a motion? It might be more efficient to now say if some of us thought that 79 should have passed, can we put it in that parking lot as something that needs to be tinkered with or improved or what have you?

Charles Irwin: We try to be fair to all of the measures. It would seem to me that we voted on it. We made a decision regarding it. I think it can be put in the parking lot. We can send it back for consideration. We can look at it with those other measures that we are going to discuss later in the afternoon. I do not think we can bring it back for voting for the core set.

Glenn Flores: No. I just meant at the end to be efficient. I do not know if it needs a motion or whether just some procedural—

Charles Irwin: I think we can do that. We can put it in that bucket for discussion.

Glenn Flores: Not to be annoying, but I am going through the math, and I was confused by the vote on number 2 because there was a 63 percent and if there were 19 people voting, I do not think that is mathematically possible. The last vote makes sense. Seventy-four percent is possible when you have 19 people voting. I want to make sure that there was not a technical glitch or somebody missed voting because that does not make sense to have 63 percent.

Charles Irwin: People could have not voted. There may be a conflict with somebody.

Glenn Flores: That is an option.

Charles Irwin: If you have a conflict, you should not be voting, and we do not know who those people are.

Glenn Flores: Sorry then to bring that up.

Charles Irwin: We are bringing up the description of this measure.

Denise Dougherty: This is one of the only child report measures.

Participant: Which one are we talking about?

Participant: It is 81.

Denise Dougherty: The Pediatric Global Health measures composed of 10 questions about a child's overall health. The questions can be answered by children 8 to 17 years old or the parents of children 5 to 17 years old. The measure produces a summary score of a child's general health status from his or her perspective. The questions asked about children's perception of their health in general, physical health, mental health, pain, friendships, family life, self-esteem, and feelings of worry and sadness. Do you want more?

Developed according to the scientific standards of the Patient Reported Outcomes Measurement Information System, PROMIS, which was an NIH program. This is meant to be comparable to the adult global health measure that is already in PROMIS. You can use this and compare adults and children.

Development was conducted with over 7,000 children and youth ages 8 to 20 and parents of children 5 to 17 across all 50 States. To have excellent reliability, validity, and additional validation is ongoing contrasting the PGHM with other measures of children's global health. That work will be completed in the latter part of 2012.

Is there anything else?

Clint Koenig: I have a question, and maybe this is the first of others that potentially have some overlap with the current measure set. I am wondering where the opportunities are here compared to the CAHPS [Consumer Assessment of Healthcare Providers and Systems]. That is measure 24 on the initial set. Is it something that we are trying to measure here that is significantly different than what is measured on CAHPS?

Denise Dougherty: Does somebody else from the SNAC want to address that? I think that is our process.

Stephen Downs: I have never been shy about sharing my opinion. I think it is different because the CAHPS is really focused on the patient experience in the health care system. This measure is really intended to measure actual quality of life. It is a little bit more distal to the actual health care experience than the CAHPS is, but in the same token, in some ways it is much more important.

Charles Gallia: I would echo that. I have had to modify the CAHPS and add in some functional health status questions in order to get some of the areas that are represented here. It was more like on the implementation side. We have this survey, and it is done. There is the magic hand of survey implementation. I am not sure about the selection practice and who would be collecting it and all those kinds of things. If I had a core set of questions that were related, that is exactly what I have done is add them to CAHPS in order to get at some of the areas that are reflected here.

Clint Koenig: I think that is wonderful, and I think that speaks to aspirational, but given the fact that we have something currently in the core set I want to look at the denominator of effort, work, and efficiency that other States have to then modify to improve what is currently in the core set to accommodate a new measure such as this.

Charles Irwin: Just for clarification, I think that decision about whether we bring the measure forward is a later decision. I think if we think of a measure as a really good measure, then we ought to vote on it, and then we ought to make a decision if we want to bring it forward because we get confused about considering what the core set has right now. I would just do that because just to make it clear how we make decisions around each measure.

Alan Spitzer: This is one of those issues that I alluded to before in the sense that, as Charles pointed out, he sometimes has to modify his questions to get the answers. This is very interesting information. I would love to know what this looks like now, then 5 years down the line, 10 years down the line. I do not think it is something that you can affect quality improvement towards in any kind of defined direct way. It is interesting information. But the practicality and the reassurance that I would need to bring this forward as a measure would mean that it really has to be applied practically to the majority at least of the population, and I do not see that happening.

Glenn Flores: I have fundamental concerns about this measure because it is essentially a health status measure. If your population is very high risk, if you had a lot of children with special health care needs, you have high rates of poverty, a lot of racial and ethnic minorities, your overall score on this may be very different from another health plan or State that does not have that baseline difference. And there is no built-in risk adjustment or case-mix adjustment. I am very uncomfortable with the notion of us just saying globally, let's have a health status measure as a measure of quality because again if my focus is underserved children, of course my health status overall score will be worse than yours if you are just focusing on the cherry-picked population. I do not think this is a good thing for us to think about in terms of quality.

Stephen Downs: I am in partial agreement with you on that, Glenn. I think it is definitely aspirational for us to be looking at measuring what happens to the health quality of the population that is being served. There are going to be baseline differences. I do not know at this point. Maybe this needs to be shelved for the reasons that you are talking about.

But things like changing scores over time are an effective way to deal with what you are talking about. I worry about shying away too much from this measure because it seems hard to affect because after all it is what we are all trying to affect. This is the reason people go into health care in one role or another is to improve the health status and quality of life of the children that we serve. I do not want to shy away from it entirely. But I think maybe a modification that interprets this as a change score might be a more effective way to do it.

Elizabeth Anderson: Can I just say that I think it's a surprise to many how many parents of kids with special health care needs say that their kids are pretty healthy. I think we cannot always assume that because there is a population that is vulnerable and at risk or even has a disability that they will not feel healthy. I personally love the idea of a child-reported measure and especially one that links to adults.

Glenn Flores: Just to respond to Steve. I agree it would be interesting if this had in fact been studied as a change score. As you well know, there is a whole science of looking at scores and looking at one score and time is very different from looking at scores over time, and most measures unfortunately in health services research are not tested for that purpose. I would assume the same is the case with this measure. I could be wrong. But that is a different question of over time and change versus I am just going to take your health plan and look at how bad the health status is for your kids versus look at this other health plan. I think that is a dangerous possibility.

Stephen Downs: I agree. Just as it is laid out as a cross-sectional point in time, it could actually have—depending on how these scores are used—it could actually divert resources from populations that need them the most. As I said, I would still argue for at least having this be a measure that we can look at in the future with some modifications.

Karen Llanos: I would just quickly clarify just because this is such a great discussion, but I think it illuminates the point of important topic area versus a measure that everyone is comfortable with. I just want to say I think, as you are considering these and thinking about them, those are the types of things you should be thinking about. Yes, this is a really important topic area, but maybe at this point in time with this particular measure and you could apply that for everything that we are considering. Just to make sure that you have both of those hats on.

Charles Irwin: I would also urge us to think that maybe this measure is not going to be a measure that we bring forward for the Medicaid CHIP population. We are actually supposed to be thinking about other measures - measures that other plans can utilize.

The question is, is the measure any good, and is the measure a good measure? I think that we have to focus on that. I would argue that maybe whether it is good for one population, bad for another population—I am not certain how we factor that into our decisionmaking at this point.

Glenn Flores: It is not so much whether it is good for one population or another, but it is a measure of quality if you have those massive differences between plans already. In other words, you are not risk-adjusting, so that is not really fair to health plans to say look at how poor your population of children's health status is, when in fact it is because they are five times more impoverished in that health plan than in another one.

Charles Irwin: I would just say that Betsy brought up a good point. I am always amazed by some of the kids I care for who I think—when I ask them how things are going, they look pretty positive about how things are going. But I would not necessarily consider it that way. This is a measure that goes at trying to understand the perspective of children and parents, which there aren't any other measures that we have had come forward that really look at that.

Stephen Downs: Given what you said, I guess I am sorry to do this, but I have to get another point of clarification here, which is what voting these things through is really going to do. If what we are saying is this is a really important topic, but it is not ready for use as a quality measure for the reasons that Glenn laid out, do we want to be voting it through at this point?

Charles Irwin: No, I do not think so. It should only be if it is a good measure.

Gerard Carrino: Before we move this discussion onto a vote, I would like to follow up on what Alan said about the actionability of this. I would like to hear from SNAC members who are involved with States or who have some kind of accountability. I would like to know from the SNAC members, how do you make this actionable as a QI intervention as a holy grail, but how do you make this actionable, as it isn't technically a quality measure in and of itself? And for any State that has done something similar or with a similar measure, what did you do with it? This is where I am equivocating between a 2 and a 3 on this one because I am not quite sure how you use it. It is definitely information I would like to know. I just do not know what I would do with it.

Naihua Duan: Can we include some annotations to the measures that are, for example, to address Glenn's question? We could recommend this measure to be used specifically to address trends and not to make cross-sectional comparisons. Each measure can be used in different ways. It might be helpful if we provide annotations to clarify that.

Charles Irwin:  Perhaps those recommendations probably would go back to the nominator. I do not know if we would bring it forward. If it does not meet a standard that is high enough, we should not be bringing it forward. We should be actually sending it back to the nominators for further work.

Naihua Duan What I mean is potentially, could there be a high standard for the specific purpose of tracking trends but not for the purpose of cross-sectional comparison?

Rajendu Srivastava: Just in response, I guess what I would say is, it is not a process measure. It is an outcome measure. Those are both quality measures. We are seeing and we will likely see lots of process measures. Just as an example, I have patients with pneumonia that I see in the hospital. Did I give them the right antibiotic? Did I get it to them within a good period of time? How long were they on oxygen? Those are process measures. The outcome measure ultimately is did the patient get better. How long were they coughing for? Things like that. And in this they are arguing—and this is the first time I had actually read it—I found it actually kind of fascinating. The developer is saying it is an outcome measure, but it is a health status measure.

It is true though that some outcome measures, as was discussed earlier, some people might argue are far removed from the purview of health care. Can you influence it? That might be true in this case. I was still also struck that it is also the only measure where we are directly asking the children what they think. I guess, as a pediatrician, one of the reasons I wanted to serve on this was to help children. I guess I am sort of struck with it; I am just wondering can we find out about what they think. It has some risk-adjustment limitations. There are some overarching concerns of how somebody might use this for not ideal purposes, but I am not thinking on that hat.

Stephen Downs: I would argue that it is actionable, but it requires a lot of creativity for actionability. I will give you a geriatric example, which is a little out of place. But health status of seniors has been shown to be drastically increased just by those seniors getting a weekly phone call. It was built on health messages, but the control group who just got chit chat phone calls, their quality of life went significantly higher. People who are willing to think out of the box and think about how you improve the quality of life for children can come up with ways that this could be addressed. It is just not as obvious as a process measure.

Feliciano Yu: I agree that this is a good measure, global health measure. One of the things that I had in mind when one chooses what a good measure is, particularly where we are in CHIPRA, and how we are trying to develop core measures at this time, is that are we able to link specific measures of quality to specific actions or specific variables that we can change in an improvement environment so that then we know the link to causality. There is the direct connection between what a process measure might be and the outcome. You pointed out an exact causal link: gave the right antibiotic, patient got better. In this case there are multifactorial, a lot of reasons why you would have a good status. That is the only thing. Whether this would be a good quality improvement-type measure, granted it is a good global measure.

Stephen Saunders: I was just going to make two points. One is we should go back to the question, is this actually reasonably feasible for State Medicaid agencies to do? In my opinion, I am not sure that it is.

My second point is, I think this measure is a good measure if it was used as a population-based measure. If, for example, HRSA [Health Resources and Services Administration] or somebody was to look into this on a nationwide basis and do a sound measure and ask everybody these kinds—not everybody, but a random sample of everybody—these kinds of questions, it would be nice to know how the health status of the U.S. children's population is changing over time. But I would not recommend it as a Medicaid or CHIPRA measure.

Charles Irwin: We are not recommending the measure now. We are basically trying to see if the measure meets our standards of moving it forward for consideration. That is all we are doing now. We are not voting to include it in the core set or on any level. We have to remember what we are voting on.

Alan Spitzer: I think this committee has to be careful not to be encouraging research. It seems to me that some of the discussion that I have heard this morning as it relates to this measure is research. It seems to me that the difference between research and quality improvement is that in research we do not know the answers to certain questions. Several people have said I would really like to know what this looks like and especially looks like over time. That is research. Quality improvement is, we have the data; it is the application of the data to medical practice or clinical practice that can provide a certain outcome. I do not think this particular measure has sufficient data to back it up, and I think we would be engaging in research to bring this forward as a core measure.

Charles Irwin: Other comments? I think we should call—

Participant: I move that we vote on the measure.

(Vote taken)

Charles Irwin: This measure will not be brought forward. Okay, the next measure.

Stephen Saunders: I have a question on this measure. I think it looks exactly like the one in the core set. Maybe the developer can explain what is different about this measure from what is in the existing core set.

Charles Gallia: When we looked at this last night, I think the difference in the measure is this one is at the provider level. Am I correct on that, Denise?

Denise Dougherty: It could be. We have comments that distinguish this—some of the SNAC members made comments that distinguish this from the other measures.

Clint Koenig: Can I ask for a point of clarification on that? I thought the burden right here is not whether to compare it to the current core set but to say, is the measure good enough to move forward.

Denise Dougherty: Right.

Andrea Benin: My one concern about this measure at least from what I am recalling and looking at the notes that I have here is that it does not have a cutoff for post dates. It is not like 37 to 41 weeks. It is 37 or later. Now, I am not sure if that is a manifestation of what the ICD-9 code is and therefore the technical limitations of the ability to do this kind of thing. It would seem to me that there would be an obvious allowance for 41 weeks and later or something like that.

Clint Koenig: I think to clarify that the initial core set measure does not have that qualifier either.

Elizabeth Anderson: One thing that I found a little disappointing was that the submitters said that special health care needs were not relevant. But I thought if the mother had a special health care need, that in fact might have been a reason for selecting cesarean.

Participant: Do we need a motion to vote?

Participant: I am still unclear on how the measure is calculated.

Charles Irwin: Do we have anyone who wants to bring this forward for nomination?

Participant: I bring this motion forward for nomination.

Participant: I will second it.

Charles Irwin: We need to vote on this.

(Vote taken)

Charles Irwin: This measure does not come forward.

Can we bring up the backward material on this one too? This is the screening and followup plan for clinical depression, ages 12 plus.

Glenn Flores: Can somebody also for each measure point out which page it is on because this is not in order. It is kind of hard to navigate.

Charles Irwin: Absolutely. I have been trying to find that myself.

Glenn Flores: 18.

Charles Irwin: Page 18.

Glenn Flores: I have a comment about this measure, and I am going to fully disclose that I am also a member of the U.S. Preventive Services Task Force, another AHRQ organization. Obviously, I cannot talk about what we have talked about where we have reviewed this, but if we can look at prior considerations of this measure. I actually did not have a lot of enthusiasm for this because if you are just talking about screening for depression an otherwise healthy population, there actually is no evidence that I am familiar with to show that it impacts outcomes in any positive way. Although I think the intention is good, there is not enough science here for us to say you need to take the time to actually go through this whole screener in every visit with an adolescent as a measure of quality.

Denise Dougherty: The U.S. Preventive Services Task Force has recommended, as you know if you are on it, has recommended screening for depression in adolescents. It was a B recommendation

Glenn Flores: Right. Part of that recommendation was the caveat about whether it actually impacted outcomes. We know that actually referring those who have issues, depression, to behavioral counseling has an impact. But my understanding was that it actually does not have an impact in outcomes to screen everybody.

Charles Irwin: Actually, the evidence is pretty clear. It actually just extends. It goes from 12 to 18 and brings down the adult measurement, which was in place. But it has a caveat. It says you should only screen if you have the resources to do the referrals. If you are not going to be engaged in that process of actively doing that, then you should not screen. And this does not do that.

Stephen Downs: Can I ask a nuts and bolts question about G codes and how pervasively G codes are being recorded in administrative claims data at this point in Medicaid systems.

Kim Elliott: In Arizona, it is pretty weak.

Stephen Downs: Having said this, a measure like this could promote integration and promotion of G codes. But I think it speaks to the feasibility of the measure.

Karen Llanos: Just one clarifying point on the issue side. This is in the adult core set. We identified it for the core set in January and then it has since been—the age range has been lowered, but it currently is part of our adult core measure set.

Stephen Downs: When you say the age range has been lowered, does that mean within the adult code set?

Karen Llanos: It is starting at 12. I believe it used to start at 18 or 21.

Stephen Downs: This would be completely redundant to that.

Karen Llanos: It is the same measure.

Glenn Flores: Just to clarify the U.S. Preventive Services Task Force recommendations, the B recommendation was for adults. It is an I statement for children and adolescents because the evidence is not sufficient. I have it right here.

Charles Irwin: For children, it is less. For adolescents, it is the B recommendation.

Glenn Flores: I will read it to you right here. It says USPSTF concludes the evidence is insufficient to recommend for or against routine screening of children or adolescents for depression.

Charles Gallia: The U.S. Preventive Task Force guidelines—I know you are aware because at the last AHRQ  meeting there was actually a little presentation I think you helped participate in about it, is the sensitivity to children in general, and having an insufficiency indication does not mean that it is not worthwhile. I am using clinical practice guidelines that are set up by Bright Futures and HRSA, also the markers and indicators that I would use as reference points that would help me engage practices and providers in keeping this on the radar. There are other metrics.

I think in terms of ACA [American Counseling Association] implementation that some of these screenings are one of the hopes that are part of a broader possible objective as well.

Diane Rowley: Can you clarify something for me please? I did not think the measure specified collecting these data only among providers who had the opportunity to refer. It was something you brought up before.

Charles Irwin: That is the U.S. Preventive Services Task Force recommendation. That is not this one.

Diane Rowley: I know, but I agree with the fact that if you screen and cannot treat, then you should not screen. My question is, did the person who nominated that exclude areas where treatment was not adequately provided?

Glenn Flores: I will update my reading because I just saw that there was an update, but it was exactly what Charles said which is the Task Force recommended screening of adolescents for major depressive disorders when systems are in place, ensure accurate diagnosis, psychotherapy, cognitive behavioral, or interpersonal, and followup. I think it maybe captures what everybody here is uncomfortable about which is that we think this is a good thing. The evidence is not completely there in terms of the outcomes, but that we know that when you refer patients who are identified, they tend to do well. But overall, the evidence is still given an insufficient grade.

Kim Elliott: From a measure perspective, I am a little concerned about the sensitivity and being able to identify those that do not have the good referral sources based on those recommendations. And I think you would have some very inconsistent results even though I do think it is important to monitor mental health, behavioral health of children.

Charles Irwin: My sense is that we should probably vote on this measure now unless there is more discussion.

Glenn Flores: This is another measure. This is what I really cannot discuss what the Task Force is doing with this because we are in the process of looking at this. I have two comments in this that I think do not prevent me from talking, which is first of all my understanding of how we measure high blood pressure in children is a purely statistical concept. It is based solely on two standard deviations, not if you were doing a range of normal studies saying let's include both healthy and unhealthy individuals and then come up with our range of normal.

More importantly perhaps also is the fact that there is no evidence that screening for high blood pressure in a healthy population of children in primary care settings actually has any impact on outcomes. My limited understanding, since I am not a nephrologist, is that there are no guidelines in general about whether any treatment in general is effective, particularly for those who are in the borderline hypertensive area. I think this is at least premature and probably not a good measure for us to hold Medicaid and CHIP plans.

Charles Irwin: Any discussion about this?

Clint Koenig: Only to dovetail on Glenn's comment, but just in the measure description it talks about the hypertension in children has been shown to be a risk factor for adult cardiovascular disease. You have hypertensive kids that become hypertensive adults who then have adults with strokes and cardiovascular disease. Extending the age spectrum beyond 18 might be worthwhile to look at this and consider this measure, not for this group, but to think through it because kids become adults.

Charles Irwin: Any further discussion?

Participant: I call for a vote.

Charles Irwin: Does anyone second it?

Participant: Second it.

(Vote taken)

Charles Irwin: The next one is sexual activity status among adolescents. May we get the actual item up here for the sexual activity one?

Glenn Flores: Someone quickly summarize what this measure is exactly. I am having a hard time remembering the metrics involved.

Denise Dougherty: The denominator is adolescents who turned 12 through 20 years of age during the measurement year. The numerator is sexual activity status current such as sexually active or abstinent, sexual activity status past, number of sexual partners, current or past diagnosis of a sexually transmitted infection, use of non-hormone-based methods of birth control such as—we know what those are, prescription for birth control contraception with an indication for pregnancy prevention, not acne or something, and current or past diagnosis of pregnancy. The intent here is really to have—for example, for the Chlamydia measure, there is a lot of controversy around the way sexual activity is actually documented. This is actually a facilitator for those quality measures that are looking to see whether a service was provided for kids who are sexually active, so screening for Chlamydia, for example, or other STIs.

I can tell you what is wrong with the current Chlamydia screening in women measure if you want. The current Chlamydia screening in women measure does not allow for an exclusion for women who are on contraceptives for non-contraceptive reasons, such as acne, dysmenorrhea, menstrual irregularities, dysfunctional uterine bleeding, pelvic pain from endometriosis, and polycystic ovarian syndrome. With information learned from testing this measure, NCQA [National Committee on Quality Assurance] intends to explore revising the Chlamydia screening in women measure to exclude women who have a prescription for birth control but have documentation that it is being prescribed for non-contraceptive use.

Diane Rowley: I would think if the intent is to get at sexually transmitted infections, then we should not have a second, this just to come up with a denominator for it. There should be a well-defined description of the Chlamydia screening, if that is the intent. I am not understanding what quality improvement we get from knowing about sexual activity just as a fact.

Stephen Downs: First, to address that I think with a clinician's hat on knowing the sexual activity status of an adolescent patient is standard of care because there are so many implications to what happens when a patient is classified one way or the other. I think it is a laudable goal to say that we should be encouraging every clinician to get that piece of information. I am a little concerned about the fact that with this measure the only mechanism for obtaining the data is chart abstraction, which is expensive and challenging, and it is described as not a double barrel, but a multi-barrel measure that there are multiple ways that you can meet these criteria, which makes it a little bit of a difficult measure. Those are my main concerns.

Diane Rowley: But I think this is an example of where this is important information for the provider, which I am not questioning at all, but in terms of how is it useful at the State level or the population level. I am not convinced that it is such a benign measure that it will be important, and it could push States in the opposite direction. To me it is more of a political issue rather than a quality improvement issue at the population level.

Charles Gallia: In addition to the fact that it involves a chart review and the kind of sampling and strategy and decisionmaking that you would have to do in order to really get into place. There is a strong component of public health surveillance, particularly the reportable conditions and the complexities. When I start to think about it as a Medicaid program going out in clinics and starting to ask for charts on this particular subject matter, we could probably go back to that previous measure about the blood pressure because it starts to get elevated. I am thinking about how that would be translated and reported in a public process.

Charles Irwin: I guess I would argue that if we were talking about quality in the adolescent population, this is like adolescent 101 minus 200. If you are not documenting about your adolescent patients starting at 12 years of age, there is something wrong with what is going on in clinical practice. And I will tell you when you look at charts because we do a lot of this. This is a lot of the work we do in our group. A lot of docs do not even do this stuff. Or if they do it, they do not document it. They do not even ask the question. But I am not that interested in all the other questions that follow from this. To me this is the basic question. I know how difficult it is going to be to get these data.

Diane Rowley: What do we know about the reliability of the data, reliability and validity of the data? Has there been a test to see how accurately we could collect the data from medical records?

Denise Dougherty: Well, that was included in the submission. I am trying to see where I would find that information because I do not have the CPCF form. Does RTI know? Is there a way to find the CPCF form so we can find that information in the proper place?

Participant: We are projecting the CPCF form.

Stephen Downs: Based on the extraction that is in here, most of the evidence that is presented is on the importance of obtaining this information rather than any metrics about the measure itself. I am sort of reiterating what I said earlier, which is I think the complexity of the issue of the measure as it is described is going to make it very difficult to make a judgment without empiric data as to how sensitive and specific it is. I am concerned about the measure itself, but I am highly enthusiastic about the topic, measuring this as a topic, because I think practice is inferior, and it is adolescent 101. I am enthusiastic about the way the measure is defined.

Elizabeth Anderson: I also could not tell if teens themselves were involved in some of the reviews and discussions. I may have missed that.

Stephen Downs: I will make a motion to vote.

Glenn Flores: I just want to support, but before we do that, about what Charles said, I was just teaching medical students about this, and this is part of HEDIS and this is just such a fundamental issue. I agree, Steve, that it is not the perfect measure. But it has so many huge implications for Medicaid and CHIP in terms of if you do not ask this and they become pregnant, they may become the pregnant mother part of Medicaid that we are talking about and introduce lots of cost.

I do think it should be a standard because you are right. It is laborious. But a lot of States and plans will start to do it if you say this is a measure. You need to start doing it. And they will figure out how to do it better. If you are talking about the adolescent population, this is if not the most important, one of the top two or three things that you need to look at in terms of quality of care for adolescents.

Stephen Downs: With that in mind, for folks who have an eye on how this process is going, to go forward, do we run the risk of inadvertently accepting an inferior measure for an incredibly important topic by moving this one forward as opposed to encouraging a much better measure for such a highly important topic. Maybe Denise should speak to that.

Denise Dougherty: I think the nominators are in the room and are hearing your comments, and we can certainly share the other comments with the nominator. They can resubmit.

Francis Chesley: I also would reiterate where you started and the guidance from the chair that at this point, you are voting on the quality of the measure. And then there is subsequent iteration of discussion that happens after you have that group of measures identified.

Denise Dougherty: But I can see you do not want to be seen as not recommending this without the transparency of why.

Glenn Flores: Procedural question. If we were to vote for measure and then there was still a general sense that it could be improved, would the project at large then pursue that, or how would that be handled? Once we vote, is it written in stone, and it cannot change? How do we interpret this?

Francis Chesley: No. This is an iterative process that you are using to get to recommendations for improving the core set. And everybody who has nominated a measure, whether they are in the room or not, will have an opportunity to get some feedback and decide what they are going to do subsequently with the measure, recognizing that they may be recommending measures to this group, but also those measures may be recommended to other groups who are looking at quality measurement for children.

Glenn Flores: This could theoretically go back to the Centers of Excellence. It could be fine-tuned until everybody is really happy.

Charles Irwin: I think, Glenn, this raises what you brought up earlier today about domains. What I hear from people is, this is an important area. It sounds like the measure is not where—several people have said they are not certain about the measure of where it is. I do not want to bias voting.

Carole Stipelman: I am concerned everybody is worried about the feasibility of chart review and we are not going to end up with any outpatient measures if we focus on that. It is so easy to get hospital data, Medicaid data, but we need some outpatient measures.

Charles Irwin: That probably needs to be something that we feed back to people. We do really need those measures.

I think we should vote.

(Vote taken)

Charles Irwin: We have two more to go before lunch. Give us the numerator and denominator on this one too. This is the dental care. Can you tell us what page it is on?

Participant: Page 22.

Denise Dougherty: This is a survey measure. I am not sure which survey, but they are all similar. National Survey of Child Health (NSCH), National Survey of Children with Special Health Care Needs. It asks parents who are contacted by phone the number of preventive dental visits the child had in the prior 12 months. It allows for stratification by children with special health care needs. I guess it is the special health care needs survey. It includes race, ethnicity, and a variety of other sociodemographic variables enabling the identification of disparities by subgroups of children. The submitter points out that measure 13 in the initial core set, which was the percentage of EPSDT eligibles that received preventive dental services, does not allow for stratification. Is that correct, Karen?

Karen Llanos: This is drawn from the EPSDT form 16 which Charles alluded to earlier. States have to annually submit that report to CMS—the percentage of individuals ages 1 to 20 who are eligible for Medicaid or CHIP expansion programs that receive the funds it is a percentage. I am not sure if the, I do not want to speculate.

Denise Dougherty: And this is for children 1 to 17 due to the nature of the survey. The other division or cut out you can make is by publicly insured, privately insured, or uninsured which again is parent self-reported in a telephone survey. I do not know that that part of the survey has ever been validated that parents are reporting correctly on their insurance.

Stephen Downs: To me there is one sort of, in my view, marginal upside of being able to stratify by special health care needs versus non-special health care needs, and that is using the MCHB [Maternal and Child Health Bureau] definition of special health care needs against a couple of downsides. One is the challenge of running a survey, developing some sort of a phone survey, having a sampling strategy, and having the infrastructure to do that. And the second one is the validation of parental report on this particular issue.

Denise Dougherty: Though the States could get the data from CAHMI [Child and Adolescent Health Measurement Initiative] or CDC [Centers for Disease Control and Prevention] for their State on this measure. The State would not necessarily have to do the survey itself.

Stephen Downs: There the loss of stratification would be Medicaid CHIP versus non-Medicaid CHIP because if it is statewide—

Denise Dougherty: You get a State average for your publicly insured population.

Diane Rowley: There is a comment here that says basically, although this is a very important measure, the measure is missing much information. I am curious to know what the missing information is because I agree that it is an important measure, but I want to make sure it is adequately defined.

Denise Dougherty: I would just take a guess that there were a lot of measures submitted by CAHMI. Each measure did not describe entirely the methodology for the survey. I think that's maybe what people meant by missing information. We did not have the resources to go back and get that information and input it into, download it into every measure.

Charles Irwin: My sense in looking at this measure was that individuals thought it was important, but there was not enough there to support moving it forward. I think that is perhaps why we see this—greater than 2.5 because there was a huge range here in terms of how people scored this. I am concerned that people were voting for it based upon we needed a dental measure. I think oral health is really important for children and children's health care needs too. I do not know if this measure holds promise right now.

Stephen Saunders: I think that maybe part of the variation might also have been the fact that Medicaid already reports on this. And they use claims data, which we know is easier to manage and is perhaps more precise than a survey. I think that might account for some of the variance as well. It seemed duplicative to consider this one when they already have one, which is the same measure.

Stephen Downs: If this were adopted as a measure, is there a liability of making the measure dependent on the continued support and execution of the CAHMI surveys. Does that create a little bit of a liability in this setting?

Charles Irwin: The other information that is a point of reference, but again it is the—the technical expert panel is meeting in 2 weeks at MCHB to determine what is going to be the content of the survey for the next round of surveys. I am not certain how we—I know we have to measure what is in front of us, but also we ought to be not totally oblivious to what is going on in the world.

Glenn Flores: Can I ask a question? I agree that this is a really important measure, and what I am hearing is there are some reservations about the way that the information is collected because it is parental report. Is that the main objection? I am having a hard time seeing why this would not be a high-priority measure.

Stephen Downs: Since I raised this issue, let me comment a little bit. It just falls outside of the kind—I am not necessarily objecting to this. It falls outside of the kind of measures that we have looking at where the Medicaid administrations would literally go out and gather that data, for example, through administrative claims or running their own chart abstractions or whatever. I have interpreted this measure in two different ways. One is that the measure would be duplicated by another process that would go out and have its own sampling strategy and make the phone calls and collect the information or alternatively that they would just gather the State-specific information from the CAHMI process itself.

To be honest with you, the way it was described in here, I could not tell which thing was actually being proposed. If it was the latter then it is highly feasible. You just go out and get the State-specific data, but it makes you then dependent on the CAHMI process to get it done. It is highly feasible until this question gets dropped or CAHMI is no longer funded or something like that. Then it becomes pretty infeasible.

Kim Elliott: I guess from a State perspective, I would be concerned about having almost duplicative measures. We have the one set already in the EPSDT CMS 416 reporting, which is very similar. It comes from administrative and claims data, and we know it is specific to Medicaid. We know it is specific to CHIP, and we know we can do something with the results of that. The survey is not as reliable but still strong. I think relying on other sources would be a bit more challenging and harder to make actionable.

Charles Gallia: We have an EPSDT expectation. There are goals that are being put forward. There is a reporting requirement by States that have these programs. It has been in operation for quite some time. If the only concern was about the ability to stratify, the EPSDT stratification is possible because it is the same data sources. As long as you have—we get duration of employment, duration of eligibility, the basis of eligibility, race, ethnicity, urban, rural—all those possibilities are there with the measures.

I have used the National Children's Health Survey in a lot of different ways, and I use it as a benchmark gauge to determine where we are when we do not have a measure yet. But I do not use it as a performance expectation that I can translate back to a managed care organization and articulate objectives or even like a little performance expectation. It is a good gauge and a good reference point that allows me to do something that I cannot do with administrative data, so it is complementary. I am just putting it out there. We have a measure. This complements, but I do not know that it would be either redundant and certainly not necessarily an improvement on what is in place.

Charles Irwin: Just raising a point of clarification for voting is that at least we voted on the cesarean measure, and I forget whether it made it forward or not, but there was a cesarean measure. Having another measure does not necessarily warrant taking it off the table or putting it on the table. I just want to be certain that we do not have the same bar for everything.

Denise Dougherty: Just another piece of context. There is a similar measure in the MEPS, the AHRQ Medical Expenditure Panel Survey. And that can be reported by private versus public, which is quite validated. But that does not drill down to the State level.

Glenn Flores: Just one last thought that this interesting conversation has raised for me, which is that I think it was Elizabeth who brought up the issue, should we be more family-centered when we think about this. And it is interesting to think about the claims data, which we say are objective and superior. But if a family actually does not think that they have a preventive dental visit and actually did, that also tells us something. I know it is a lot of literature and development screening about the fact that it is not just doing it, but telling the parent and addressing any concerns. I think this measure is interesting in that light because it does look at that, and you could compare it to claims. And the other interesting thing is it is from the NSCH so you would have national data to compare, which we may not have with a lot of the other measures. I think there can be some arguments for this actually being a reasonable thing to think about.

Charles Irwin: Can we vote?

Elizabeth Anderson: I place a nomination.

(Vote taken)

Charles Irwin: We have one last measure—surgery.

Charles Gallia: I was hoping that—I am looking forward to the development of measures that are parallel to the prevention quality indicators for the adults that would be reflected in the CHIPRA core set. This one I thought could have the potential of being one of those measures. It is obviously not a preventive quality indicator but an outpatient safety one.

I actually ran the results at a State level to see how common an occurrence it was in Oregon. In an 8-year period, we have 28 mortalities that it could not connect up with a delivery issue or a genetic exclusion that would be considered as part of that. And then I asked myself, what would I do at a State level about those? If it is a volume-based proportion because the procedures—there is not the skill set that is engaged in what the activity is. As a policy initiative, as a State program, I am going to try to dissuade areas from doing heart surgery. Am I going to try to consolidate, realign those. It is a little bit problematic for me in that regard. Not that I do not think it is important or that the data do not exist, and I can monitor. From a program standpoint, from what am I going to do in terms of facilitating improvements or not, that is my concern. They had exclusion criteria that—

Denise Dougherty: One thing to do is preferentially pay for surgeries in high-volume, low-mortality settings.

Rajendu Srivastava: I just wanted to mention again that this is more of an outcome measure. I think the creativeness of how to use this would be the way to go if you look at the pediatric heart mortality networks and stuff that has happened in this country over the years. People have moved their patients to try to maybe locate them in that sense. I actually do not know how an individual State may or may not take that on. Health plans may want to take that on. Not looking at it from the point of is this better for a State or better for a health plan or better for somebody else, just thinking about it. Is it a good measure? I thought from an outcome measure point of view, it does have some of the attributes I think we are looking for.

The concern I have, and this is more process, as other people have commented you cannot look at mortality without looking at the volume. That might actually be true. But they submitted it separately. I know we were voting on them separately, but it might be that one of our strong comments is just combine them if that actually is what you need to make the measure.

Andrea Benin: I would agree that this is obviously a very important topic, and it is tied often to volume. There is a registry done by the Society for Thoracic Surgery that most reputable places that are doing these kinds of surgeries participate in. It has a risk-adjusted mortality metric in there that is considered I think the industry standard, if you would. I would venture to say that that is actually the mortality metric for pediatric surgery that should be used nationally. It is a reasonably robust registry as far as these things go. But it is actual data that you collect from a chart and do all this. My understanding is that virtually all of the places that do heart surgery, congenital heart surgery, are participating. It may not be entirely 100 percent.

If this were something that the Medicaid programs were really interested in, it would probably be better to create some kind of structural metric around you have to participate in it. The places in your State that do this have to participate in these registries or something along those lines. I think that would get at this in a much better fashion. I do not see the point of collecting this using administrative data. It does not make sense to me. That would be my take on that.

Denise Dougherty: Sometimes those registries by professional societies are proprietary, and sometimes they do not collect data by the payer.

Andrea Benin: On the adult side, one of the structural core measures is that you have to participate in a registry for some of these kinds of things. That is how they have gotten around that. I think that those are options. That is not what is on the table here. I am just saying there is an alternative sort of way that people typically look at this other than using administrative data.

Glenn Flores: I have a concern about this measure. On the first pass, I think it is an interesting concept. But then if you think about how we do referrals for major congenital heart lesions, you send them to the centers that take care of the most seriously ill kids. If you are one of these centers, your mortality rate is going to be very high because you handle the riskiest, sickest kids. Just prima facie saying we are going to look at mortality rates, we do not care whether you do 10 of the most intensive bypass procedures or whether it is just a simple valve replacement I think is disingenuous. I would be very concerned. I tried to wade through this and see what sort of risk adjustment there was, and I did not really come away satisfied. I do not know about you. I personally think from the inpatient perspective I would be very concerned about trying to measure a major academic center that does hundreds of these procedures versus one in a rural area that does one or two and probably will do great because they take care of the least sick children.

Rajendu Srivastava: Since you asked—it has been a while since I read it. I am trying to pull it up. It says risk-adjusted rates of in-hospital deaths. Glenn, if I recall correctly, they use a lot of the—there is a group of very well-known cardiac health services researchers that for a number of years have been using ICD-9 codes to come up with appropriate risk adjustment for complex cardiac surgery and actually have different stages. They have actually worked with the thoracic surgeons, a bunch of cardiologists, people at AHRQ. They have come up with this and published a fair number of studies that have pretty reasonably adjusted depending on the lesion, et cetera.

I actually think, my recollection of this, and I have to look, is that this is what they were going to be using to account for that. That actually went a long way for me to say that is actually the state of our health services research, pediatric cardiologist, cardiothoracic surgeons, hands down say that is how we would want to do it.

Denise Dougherty: There is a list of denominator exclusions in the submission that was made.

Glenn Flores: That would be an important piece of information. Is that in there because I did not come across it.

Denise Dougherty: It is under exclusions. It is not under a category called risk adjustment. If you look at this, you wonder what is left.

Glenn Flores: Actually, that is a good question. I am unclear what mortality they are looking at. Which conditions are they actually going to be looking at because it excludes?

Denise Dougherty: They have a list of denominator details, and there is a whole bunch of them under congenital heart disease procedures. Nonspecific cardiac procedures 2P, congenital heart disease diagnosis 2D.

Stephen Downs: In general, I think this meets all of the criteria for being a very good measure, with one main concern. One is that it applies to a very small fraction of the population. And then the corollary with that is whether there are statistical challenges in being able to meaningfully measure improvements. Those would be my only concerns. Otherwise, I think it exactly meets the criteria of a good measure.

Clint Koenig: I think to echo your point, I agree with you. But I think in terms of where we are looking now, is it good enough to move forward? I think to your point is yes. I probably would vote yes for this in this round and probably vote no in the next round.

Denise Dougherty: Which means it would be recommended for other public and private programs.

Glenn Flores: I guess I am still not convinced for us because looking at the list. Basically, what they have done is they have taken out some of the more severe conditions. But within each condition there is severity of illness—there is also comorbidity that you need to adjust for. I am worried that again if you are the academic institution that does 50 valve replacements and valvulotomes compared to one that does one, of course your mortality index will be higher because you are doing more of them. You were talking about volume in the relationship with volume.

Stephen Downs: In actuality, the experience in the adult world is that it is just the opposite. The programs that do lots of procedures and do the most complex procedures develop the best systems for patient safety in reducing mortality.

Glenn Flores: No, I agree. But I am saying as Charles said if you have one and that one could die that is 100 percent. And if you have one and the kid survived, that is 0 percent. It is sort of a catch 22 if you will.

Stephen Downs: That is the statistical concern that I was raising. If the numbers are very small, the statistical leverage of every case becomes inordinately high.

Alan Spitzer: I am not sure that is quite true. You are talking about half to 1 percent of all children born have some form of congenital heart disease. It is not that tiny a segment of the population.

Stephen Downs: The mortality rates are sufficiently low. If you have 28 cases in one State in 8 years, you have a statistical issue.

Alan Spitzer: I agree. For example, in New York State I know they have a wonderful tracking system for the surgery of patients with congenital heart disease. I think it has come out of the fact that the Department of Health in New York State has tracked this for a number of years. And if you are not doing enough cases and your mortality inches up too high, they withdraw your certificate of need, and you cannot do surgery anymore. I think that that is an important public health issue and one of the reasons for validating this measure.

Charles Gallia: One of the levels of aggregation or attribution is important to consider and some of the measures. In one sense, it might have some relevance for facilities, but probably not as a State measure. With that, I am going to move that we vote on it.

Rajendu Srivastava: Having published some of these studies on mortality and heart disease, it gets tricky I agree, depending on how you aggregate it. We have done it actually using some national data, the KID [Kids' Inpatient Database] data that AHRQ works on and looked at it different ways. I agree. You have to have your questions up front, but that is research.

The second point though is, Glenn, looking at this list. The list I am familiar with is a risk stratification that is commonly used in pediatric heart mortality that actually has different levels of complexity based on what the child is born with. Are they a 1, 2, 3, 4, 5, 6? And that is considered well done. Rereading this I cannot see that in here. I do not know if that is what they are using. It very well could be, and I am just not familiar with it.

Denise Dougherty: It is all from claims data. You would not be able to tell what the child was born with, I do not think, from claims data.

Rajendu Srivastava: No, but the health services research person who did this was Kathy Jenkins. She went through a several year process in which the group of surgeons, cardiologists from around the country risk, stratified based on the ICD-9 codes from administrative data and did some pilot studies looking at that, probably I am guessing some validation. That is considered routine risk stratification. If you get a child with a VSD [ventricular septal defect], that is risk stratification one. If you have a child with a very complex hypoplastic left heart that is all the way to 6, I do not know if that is what they are proposing in here.

Clint Koenig: To respond to that Glenn and I were talking here, I do not know if it would be helpful to put up the risk adjustment methodology variables from the full report. They do talk about a RACs one risk category, and they talk about reporting on that and other variables. It is probably below that. It is inception two detailed measured specs. It starts with risk adjustment type.

Glenn Flores: Looking at that, that would be wonderful if every State and health plan was able to do that. I doubt that they are going to do a generalized estimating equation and have the RACs one risk category data. This would be I think a reasonable approach. Are we putting a really high bar for States to be able to do this? Is it practical?

Denise Dougherty: It goes into the technical specifications that are used. I do not know how you audit that.

Karen Llanos: Our past 2 years' experience have shown that just because it is in the text specs does not mean it can be executed by States consistently or even in full. I think there is something to be said about a methodology, whether they are sampling or data collection, that is maybe not as complicated or something that would be more feasible. States range in their data capacity, so it is hard to tell, but I just wanted to highlight the fact that the specifications are just the starting point, and then it is the implementation, which is really difficult.

Denise Dougherty: Remember though we are not voting on whether it goes to the improved core set right now. Just on whether what you have in front of you is a good measure.

Glenn Flores: I hope you do not mind, Kim. I just asked Kim here from Arizona, do you do that, and she said no, we could not. I am not sure whether we are supposed to vote on the feasibility. I would bet that a lot of people do not even know what the generalized estimating equation is that you use it for. That does not mean you cannot do it. That would be my only worry. They are doing the right risk adjustment, but it is fairly complicated, and I am not sure there are going to be many States who are going to say we will devote our biostatistical effort and pay for the biostatisticians to do the GEE [generalized estimating equation] and put all these variables in and come out with this.

Naihua Duan: The software for GEE is pretty widely available. I would guess if the State is willing to do it, it is not a prohibitive to try to get a good biostatistician to work with them.

Stephen Downs: If CMS or AHRQ wanted to have SAS data shared, SAS files shared among States, you could overcome a lot of that.

Denise Dougherty: But we are not voting on the State use right now. We are voting on whether it is a good measure.

Charles Irwin: Are we ready?

Elizabeth Anderson: I second the motion.

(Vote taken)

Charles Irwin: We are going to adjourn for lunch. We will regroup at 1 pm. Where is the lunch?

Participant: It is where we ate last night. The lunch is in the restaurant, which you have to pass on your left to go to the restrooms.

Return to Contents
Proceed to Next Section

Page last reviewed March 2013
Internet Citation: 2012 Meeting of the Subcommittee on Quality Measures for Children's Healthcare: Transcript (continued). March 2013. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care