Genetic Tests for Non-Cancer Diseases/Conditions: A Horizon Scan
Table 1. Invited Peer Reviewer Comments
|Reviewer1||Reviewer Comments||Author Response|
I apologize for the hurried review. I did spend some time on the topic evaluating completeness and have the following comments that might prove of some benefit. I commend the authors for clearly defining the strategies and presenting evolving methodologies for dealing with a difficult subject.
Although the document clearly states that it is an update, I found it difficult to evaluate without collecting additional resources (e.g., the original report, online material). I suggest that, in the future, the material be made more stand-alone. Also, it may have been useful to allow reviewers to access the DB of potential relevant tests.
It is interesting that the reviewers did not attempt to identify results from other horizon scanning (or horizon scanning-like) reports or postings. I'm sure there are several out there, but I know of the one posted on the EGAPP website (http://www.egappreviews.org) and thought it might be interesting to determine which of the potential review topics might also be included in the report under review. That list is not meant to be complete, but should form a reasonable validation set. I first removed any cancer-related tests/scenarios along with those relating only to children. The resulting list of 27 tests/scenarios is contained in Table 1. I then searched both the original and revised AHRQ reports as well as the www to determine overlap. If the test/scenario was present on the EGAPP list, but not in the reports, I reported the results of my search. Of the 27 tests/scenarios, four were in one or the reports (e.g., HFE testing), and 3 were not found anywhere (indicated by '???'). I found listings for the remaining 20 on the www (often from clinical laboratories) that I could not find in the AHRQ report. Perhaps I am have misread the reports' charge or did not get a complete listing of topics - if so, I apologize. Perhaps it is more likely that these tests/scenarios were identified via the horizon scanning, but were left off the list because they were not considered relevant. If so, it is not clear why they would be removed. Would the authors be able to explain why at least some of these tests/scenarios were not identified and/or included in the report?
It appears that tests that are offered over the internet would not be included. Several of these sites use CLIA-certified laboratories and require test (at least select one) to be ordered by a physician. Why would these not be suitable for inclusion?
To ensure clarity and continuity, we will be including tables from the previous report of the horizon scan report in the appendix section.
We reviewed the list of tests provided to us by the reviewers. We have included additional tests that met our eligibility criteria.
With expanding lists of tests, it is difficult to compile all of them in the main body of the report. However, we will make a full list of tests available in the 2007 report to be available appendix of the report.
Some of the tests identified in the list pertains to cancer, and a few were available in the previous report. We have included the ones that are clinically relevant from the list provided to us.
We verified the tests indicated in your table. The majority of those tests was available in our previous report and was available either in table 1 or 2. In the prior reports, we have excluded internet-based tests, where test samples are not collected under the supervision of trained personnel, and where the authorization from physician is not mandatory. However, in consultation with CMS and AHRQ, we have expanded our scope of the horizon scan to include internet-based tests that at least mentions authorization by a physician.
Our expanded inclusion criteria should adequately address this issue.
This report updates the 2007 horizon scan of genetic tests for non-cancer conditions commissioned by AHRQ and published online at http://www.cms.hhs.gov/determinationprocess/downloads/id49TA.pdf [PDF File, Plugin Software Help]. The Introduction (almost unchanged from the previous version) defines genetic tests according to the Final Report of the Task Force on Genetic Testing, Promoting Safe and Effective Genetic Testing in the United States (http://www.genome.gov/10002405), which is now more than 12 years old. The Task Force definition limits genetic tests to "heritable disease-related genotypes" and specifically excludes somatic mutations. Using this definition (even for non-cancer genetic tests) could limit consideration of newer techniques, such as gene expression, epigenetics, and proteomics. The Google alerts used for horizon scanning may need to be modified to capture such tests.
The approach used for horizon scanning (Google alerts combined with targeted searching of selected laboratory websites) is well-justified. It would be interesting to know how the results compared with those of the more exhaustive approach used in the previous report or whether other Google alerts were considered. The report should specify the period of time when the search(es) were done, especially because (as the authors point out), many web links are ephemeral. A cross-sectional sweep of the Internet every 2 years will probably identify tests with "staying power," although the information in the report may quickly become outdated.
The tests in this report are divided into pharmacogenetic vs. non-pharmacogenetic tests, with 8 found in each group. The non-pharmacogenetic tests are a mixed bag, with uses ranging from diagnosis of Mendelian disorders with late onset (e.g., dilated cardiomyopathy) to more speculative "predictive medicine" (e.g., general nutritional assessment). Many other tests in the latter category have come and gone and many others are available in the marketplace. How did the authors choose to include this one? In designing the database, it might be helpful to include one or more additional variables to classify tests more generally into categories according to their anticipated use—e.g., risk prediction, screening, diagnosis, prognosis and management, and pharmacogenomics—especially if the database will include cancer-related tests.
The Results section refers to Appendix I, which must be the same as Appendix A. The title of Table 3 is "Topics for which a focused review of pharmacogenetics was conducted or is currently in progress" but it isn't clear what the review process is. If a review has already been completed and published, a citation should be provided. If reviews are being conducted via another AHRQ process, citing it here would be helpful. The last 2 columns in this table should be labeled to indicate that they contain numbers of Medline abstracts.
The meaning of Figure 2 is not entirely clear. Does the tall bar centered at zero mean that no relevant Medline abstracts were found for at least half of the 16 tests? Is this an expected finding? Does it reflect on the sensitivity of the Medline query? The figure legend states, "The plot shows that most of the newer tests, especially the pharmacogenetic tests in clinical use had less than 150 citations in the Medline identified through a preliminary search." However, it isn't possible to distinguish the pharmacogenetic tests in the figure and the threshold of 150 seems arbitrary: it doesn't correspond with bar divisions or apparent median. The most important finding to explain here is the first bar.
The one-page summaries in Appendix A include a lot of useful information. More specific Web citations would be useful (e.g., specific pages on medicalnewstoday.com and genelex.com Websites for CYP 2C19 testing for resistance to clopidogrel). For ease of use, I would suggest changing the order of headings to the following: Purpose, Diseases, Clinical Uses, Specimen, Methodology, Availability, Sources, Medline Searches. This would group clinical and laboratory-related information together.
The charts of "Medline hits" are dramatic but they take up a lot of space. It might be more helpful to show these data in a table. It would be easier to make comparisons among tests (e.g., to see which ones might have sufficient data available for a systematic review) using numbers. Vastly differing scales on the charts highlight trends while down-playing the actual numbers of citations.
Google alerts is one of the many tools used to evaluate the grey literature. Yes, we acknowledge that it is ephemeral to search the internet. The hits from google or other internet sources are used initially to generate a list of potential tests or the manufacturers. To clarify the grey literature searches conducted in sites other than google, we have added a brief description of other websites searched by us.
Thank you, we will expand our search terms to include all the newer techniques.
Currently the search date and the website access date are maintained in the database. "A cross-sectional sweep of the Internet every 2 years will probably identify tests with "staying power" is beyond the scope of this report.
If the test is still indexed as offered at a manufacturers website or a laboratory site, we include them in the database.
We do include the anticipated use as the "Purpose of the test".
Thank you, we have added references to the completed systematic reviews or technology assessment reports conducted by Tufts EPC in non-cancer conditions.
The histogram generated by a statistical program is confusing to the readers and we have replace this with a simple table to ease interpretation.
The sites that we gathered information on test sources are available in the section "sources."
The figures of medline hits are provided to get a visual picture of changing publication patterns over the years. We have placed the one pagers in the appendix.
Thank you for the opportunity to review this fairly thorough and interesting report that provides an update on novel genetic tests for non-cancer diseases and conditions. My comments are as follows:
The authors begin appropriately with a review of the terminologies and definitions. It is noted that they searched for changes in the existing definitions and found none. This is relevant and appropriate for this type of report. The population of interest is well-defined.
It is not clear how the authors define gene-based biomarkers, which they claim to have excluded, because certain pharmacogenomic tests can be considered to be gene-based biomarkers.
Although I appreciate the fact that conducting a search that includes the published scientific literature as well as the grey literature may be resource-, time- and labor-intensive, I believe that to be thorough it is important to also systematically search the published literature also. Perhaps the resources of PharmGKB might serve as a good starting point.
Although the authors mention, on p. 12 of the report, that FDA approval status for each test is included in the "Gene Test Tracker" database, I could not find any evidence of this in the table and the results detailed in Appendix A. Since this report was requested by the Coverage and Analysis Group at the Centers for Medicare and Medicaid Services (CMS), the regulatory status of each gene test would be useful to know.
The authors mention that, with their new relational database (Gene Test Tracker), it is possible for users to add new genetic tests "by simply clicking the add new button." Although I appreciate the value of public data-sharing and the value of this new resource, I wonder whether there are provisions for monitoring the accuracy of the information that is entered.
Finally, there are a numerous typographical and grammatical errors. I would suggest that the authors thoroughly proofread the report and correct these errors prior to publication.
We have included the new SAGCHS definition of gene tests and we have edited this section.
As part of conducting focused reviews, we do search the published literature systematically. It would be difficult to do searches in published without focused questions.
Yes, we have added that information.
The database is currently available as a resource to CMS and being used for internal purpose.
We have proof-read the document
The concept of developing a horizon scanning approach/method for genetic tests for cancer- and non-cancer-related diseases/conditions, with biannual updates, is a very important addition to information on emerging tests. Of course, a first step in horizon scanning is to identify your target audience - who is it that you intend to inform? In this case, the primary audience is the commissioning body, the Centers for Medicare and Medicaid Services. As stated in the 2007 report, CMS is seeking a "ready reference for discussions in this area" that includes basic information and an estimate of the potential literature available on each test. The question is whether AHRQ would also want such reports to represent an important resource for other policy makers, advisory groups, and test evaluators. Towards that end, improvements in organization and clarity are needed. Most importantly, the 2009 report (and future reports) will have much more value as a stand-alone document.
The key results of the first scan in 2007 (Table 1 - 90 tests) are not dealt with in the 2009 report. The 2009 report adds 16 tests, but few readers will want to track down the previous report to find the initial 90 tests identified. In addition, it seems both CMS and other readers might benefit from knowing what transpired with these 90 tests in the intervening two years. Are all still being offered clinically? Have some achieved FDA clearance/approval? Have systematic review/HTA reports been released on any of the tests?
Inclusion/exclusion criteria are stated and appear consistent in the 2007 and 2009 reports, but are threaded through the text - a clear list would be very useful. For example:
Pg4: Notes that the authors excluded "gene-based biomarkers since the aforementioned definition excludes somatic and protein-based tests" The definition cited is from the 1997 Report of the Task Force on Genetic Testing. The authors may want to consider much deliberated and updated definitions that are more consistent with current technology development. From the 2006 SACGHS report on Coverage and Reimbursement of Genetic Tests:
Pg 5: The categories of test applications described are useful, but do not seem to be applied either in the Tables or the test summaries?
Pg 6: The rationale for focusing on grey literature sources is made clear in the 2007 and 2009 reports, and is consistent with findings of other horizon scanning programs. However, the actual scope of the searches in 2009 were not clear to me. The 2007 report has a clear list of grey lit searches on pg 7; along with detailed descriptions of the use of sources (e.g., GeneTests, LexisNexis and 14 others; pp. 8-14). The section on Grey literature search in the 2009 report mentions only Google News. Were the methods the same as 2007 or more limited?
Pg 6: The beginning of the last paragraph states that they did not contact commercial laboratories directly for any additional information. 1) Two paragraphs above, they state that "we also visited the relevant laboratories that appeared in the news items" - do they mean they visited their web sites? 2) In the 2007 report they mention reactive (stakeholder input) and proactive (searches) approaches. Was the reactive approach of the methods for 2009, or was it not productive and dropped? Were conference proceedings reviewed?
Pg 7: 1) Table 1 in the 2009 report lists 20 websites systematically reviewed looking for new tests, and the text notes that "several new websites are added to Table 1." For those interested in knowing what is knew, it is necessary to search the 2007 report (in which Table 1 was the list of tests) to find a list of 10 laboratory websites on pg 10 of the text. Again, the information is there, but connecting the dots is time-consuming. 2) At the top of the page, it is noted that websites offering only DTC tests were excluded from Table 1. Does that mean that any tests offered DTC are excluded? Many tests offered DTC are also ordered by clinicians - this is confusing.
Pg 14: Tables 2a and 2b list newly identified tests. Not only are the 90 tests from 2007 not provided, but the formats for Table 1 from 2007 and Tables 2 from 2009 are different. The 2009 report should include a table of all 106 tests in one table, with updated information on the 2007 tests as noted above. Using the GeneTestTracker table format would be ideal.
Pg 15: 4 of 7 entries in Table 3 are cancer-related. It is certainly reasonable to include PGx tests related to cancer drugs, but that should be made clear in the inclusion/exclusion criteria.
Pg 10-12: The Tufts GeneTestTracker is a great idea and much-needed.
Pg 23: Test summary for CYP2C9/VKORC1 for warfarin dosing/sensitivity. It seems the CMS and others could benefit from knowing when review and appraisal has taken place re a topic:
The tables from previous reports will be available in the Appendix section of the new report.
The survival trends of these tests are beyond the scope of this report.
We will summarize in one section the inclusion and exclusion criteria
Thank you, we will add the new 2008 SACGHS report on Oversight of Genetic Testing definition to our report
We have made an effort to include consistent terminologies in the one-page summaries. They appear under the section "Purpose"
We continue to review grey literature search similar to the methods described in our 2007 report. Our main focus and yield in recent years have been genetests.org, Google News, major laboratory websites and manufacturers' website.
We will merge some sections of 2007 report into this updated report to give additional clarity and continuity. However adding the tables from the prior report We have excluded internet based tests, where test samples are not collected under the supervision of trained personnel, and where the authorization from physician is not mandatory.
Thank you, we have updated our new report per the suggested edit
This table just details the information on focused review conducted to date. We will exclude cancer tests as these are confusing to the readres
Currently, the database is in use for internal purposes.
Around the same time as the ACCE review, the CMS through AHRQ assigned us (Tufts EPC) a review on CYP2C9/VKORC1. We will reference the Tufts EPC technology assessment report.
Thank you, we have added these sites to our list of websites for grey literature searches.