Skip Navigation U.S. Department of Health and Human Services
Agency for Healthcare Research Quality
Archive print banner
Tools for Monitoring the Health Care Safety Net

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Integrated State Data Systems (continued)

Components for Successful Data Integration

Impetus for South Carolina's Integrated Data System

The need to answer complex policy questions was actually the catalyst for the integrated data system in South Carolina. In 1992, the following systemic and population-based policy issues facilitated data sharing activities:

  • Improving access to health care services.
  • Containing health care costs.
  • Maintaining or improving existing quality of care in a cost-effective manner.
  • Enhancing informed decisions in the selection of health care providers, facilities, and services.
  • Determining the appropriate types of health care services needed for the State's growing elderly and disabled populations.
  • Determining the effect of lifestyle, social, environmental, and genetic factors on health.
  • Evaluating and improving the types of treatment being provided in a wide range of settings.

The data plan that was developed took each policy issue and identified the types of data needed for analysis. This information was then cross-referenced with data and systems that had been previously inventoried. This aided in determining which datasets were readily available and what changes would have to be made to existing systems to answer these policy questions.

Given the complexity of the policy issues, it became clear that the integration of data would be the only meaningful way to proceed. Integration of data would permit tracking cohorts from providers' offices through emergency departments and inpatient hospitalizations, as well as tracking clients across a range of State agency-based programs.

During the planning process, it was determined that the collection of personal identifiers was necessary to track populations across time and service providers. Without these identifiers, there was no way to link multiple datasets from multiple sources. Using identifiers to create a key linker or unique identification number allowed South Carolina to reach beyond "silo-oriented" data systems to achieve an enterprise integrated data system that contains rich information for all areas of health care and the public sector. An example of this is a study conducted that examined health service utilization 6 months suicide was committed. It was determined that more than 30 percent of people who successfully committed suicide sought care from an emergency department or community mental health center 6 months before taking their own lives. This level of analysis would not have been possible without using the identifiers to create a key linker to integrate multiple datasets.

Technical Issues

Because of the ever-evolving nature of hardware and software, it is imperative that an integrated system not be built based on technical availability. Rather, it should reflect a research philosophy that transcends technology. There are several rules that South Carolina has adopted in developing the technical components of its integrated data system.

Technical Foundation for Integrating Data Is a Successful Key Linker System

South Carolina began "unduplicating" all personal identifiers on 1996 records. Each unduplicated person is randomly assigned a number generated by a computer algorithm. The number is not affiliated with any identifier associated with the individual, e.g., Social Security No. or date of birth. The number remains with that individual for all subsequent service use, regardless of data source or service provider. The algorithm accounts for misspellings, aliases, and name changes. For each additional record, public or private, submitted to the Office of Research and Statistics, a comparison is made to the "unduplicated" person file. If that individual is found, then the designated key linker is assigned to that episode of service. If that individual is not found, then he or she is added to the unduplicated file and assigned the next available number. To protect confidentiality, an individual's personal identifier is never associated with the service received. The final statistical file has no personal identifiers, only the key linker.

Develop Expertise in Dealing With Agency Datasets by Establishing Key Agency Contacts

No one knows the data better than personnel who work with the original data sources. It is important that the integrated data staff maintain an ongoing relationship with the key agency data staff. Administrative data systems evolve to meet the changing needs of the programs and services they administer, and the learning curve for the integrated data staff member never ends. A strong working relationship between the integrated data staff member and staff from the data source is essential.

Be Prepared to Deal With Longitudinal and Sometimes Duplicative Data

The planning process should include anticipating how to deal with longitudinal and duplicative data. Only by developing protocols for dealing with these issues will it be possible to present an accurate picture of the population studied.

Be Prepared to Address Transiency Issues

It has been South Carolina's experience that many of its lower-income residents do not reside in the same place for very long periods of time. A sophisticated GIS system can aid in tracking the movement of various populations. In structuring an integrated data system, it is important to build in an address-matching software component that compiles an address history file that allows tracking geographic mobility over time. This is especially helpful in studying environmental linkages to certain health conditions such as lead poisoning and asthma.

Understand the Complexities Surrounding Agency Data Systems

It is not enough to understand the data that agencies and organizations share through an integrated data system. It is equally important to understand the data policies, computer hardware, and data utilization of each agency or organization. This knowledge will enrich the integrated data system's ability to analyze data.

Policies and Data Systems Change Within Agencies

Have a plan ready to address change in data policy and data systems. As data elements, formats, and variables change or are deleted, the successful integrated data system will experience little disruption because a plan has been implemented to absorb such changes.

Return to Contents

Organizational Structure

The importance of a sound organizational structure cannot be overemphasized. Locating the integrated data system in a neutral organization diffuses power issues and allows data to be analyzed in a non-competitive and apolitical environment. In South Carolina, the Office of Research and Statistics (ORS) is responsible for integrating data and is located in the Budget and Control Board of the State government. The Budget and Control Board, which reports to both the Governor and Legislature, is a neutral body that does not provide services, nor does it oversee any services or other agencies. By being a neutral agency, ORS provides agencies with data that empower them to better serve the State's residents.

Building Trust

Depending on the environment, one of the most challenging components of an integrated data system is establishing trust among the various individuals and organizations contributing data. Data can personify power to many; thus, sharing data in a hostile environment can at best create insecurities, if not resistance, among data partners. South Carolina has spent decades building trusting relationships with data partners, and as a result has data from virtually every State agency, the private health care sector, many not-for-profit organizations, and medical clinics. South Carolina follows a set of general principles for building trust:

Do Not Try to Replace Existing Agency Functions

It is best to augment what is already occurring so that no one feels threatened. Most agencies and organizations have in place statistical and research teams that support their respective data efforts. When integrating data, it is best to assure the data partners that their existing functions will not be interrupted, and that the new integrated data system will not replace the work they are currently doing. It is best to identify ways in which the integrated data system can aid in or augment the existing work. The fear of being replaced is sufficient to sabotage the integrated data effort.

Each Organization (Public and Private) Is Treated Equally

Rules for partnering in an integrated data system must be consistent among all partners. To provide any partner with preferential treatment erodes the very foundation upon which the system is built. Activities such as data access or exchange should be consistently applied across all the partners. To do otherwise creates a competitive environment that also may sabotage integrated data efforts.

Steer Clear of Egos and Political Issues

It is never wise to use the integrated data to harm partners. This is a philosophy that must be agreed upon by all partners in the initial phases of development. Using the integrated data as a weapon to do harm creates an atmosphere of threat, which results in the diminished capacity of the system.

Never Criticize an Agency/Organizational Data System

Criticism of a partner's system generates feelings of insecurity and inferiority, resulting in a partner's resistance to sharing data. The omission of a partner's data can have substantial effects on the overall value of the integrated data system.

Make the Data Extraction Process as Non-intrusive as Possible

Complementing rather than competing with a partner's existing system for data extraction creates a natural tendency for that partner to share data through the integrated data system. Participation in the system should be effortless for the partners. Creating additional work for participation generates resistance.

Do Not Duplicate or Compete With Agency Statistical Offices

An integrated data system should empower the agency with robust data to further its mission and reach its constituents. Coming between an agency and its funding source, governance, or clients will have a detrimental effect on the integrated data system. Taking attention away from the partners will sabotage the system.

Gain Support of Statistical and Information System Offices First

These individuals will be working intimately with the integrated system. Have their complete buy-in and support for the effort before approaching those at the executive level. These staff will be the best people to advocate to the executive level for participation in the integrated data system.

Secure Support for Data Sharing and Linkage First

Do not attempt to work out all the issues at the beginning. Given the many regulations surrounding privacy and confidentiality, it would be easy to lose focus battling over the finer points of the system. These types of discussions can bring an integrated data system to a halt before the first data file is loaded. It is imperative to reach a mutual agreement on the philosophy of the integrated data system first and revisit it often to reinforce a cooperative spirit. The details will work themselves out as long as everyone is in agreement on the goal.

Put It in Writing

Execute memoranda of understanding/agreement establishing rules of data sharing and assuring agency control. Nothing demonstrates trust more than a written commitment. In an integrated data system, the partners should retain control over their own data. They should provide direction on what types of linkages should occur. Putting this commitment in writing assures trust. In South Carolina, partners have learned that the more they share, the more they get back in return. Integrated data systems are an exponential investment in knowledge.

Rules for Privacy, Confidentiality, and Data Access

Legislation is a mechanism for securing data sharing through an integrated data system, if it is pursued as a united effort by the partners. However, without the support of the partners, the system will not be successful, even if partners are mandated to share data. In South Carolina, original legislation addressed only private health care sector data. The legislation established the South Carolina Data Oversight Council, which is comprised of representatives from the private sector (hospitals, physicians, and nursing homes), public sector (Governor's office, Health Department, Health and Human Services Department), third-party payers, and the business community. The release of the private health care sector data is provided at the Council's direction, not that of the ORS. This reinforces one of the rules for building trust by treating everyone equally. This impartial body reviews data requests and releases data based on criteria as established in their policies and procedures.

Historically in South Carolina, release of public or State agency data is dictated by the agency providing the data through a Memorandum of Understanding or an internal data review committee or both, depending on the policies of the respective agencies. If a data request requires links of multiple datasets, ORS facilitates the requestor's proposal by bringing the partners together to review the data request. Previous experience has demonstrated that this fosters a collaborative spirit and maintains the partners' communication about and awareness of current research. It invariably results in a positive experience for the requestors and data partners.

In 2002, a proviso was passed to solidify data sharing activities among State agencies in South Carolina. The State law pre-empts HIPAA for collection of data. Key requirements include:

  • Agencies collect and provide client data to the Budget & Control Board's ORS as a neutral organizational structure for integrating data.
  • ORS establishes Memoranda of Understanding with each agency that specify confidentiality, release of data, etc.

Agencies retain ownership of their data, and no data are released by ORS without the express permission of the agency. The key linker system, which de-identifies individuals, fosters adherence to all Federal and State laws and regulations pertaining to confidentiality and privacy.

In addition to the laws and regulations specific to the integrated data system in South Carolina, there also is a statutory environment that facilitates the receipt of data from agencies. South Carolina laws permit the sharing of data, with identifiers, to registries such as those for cancer and reportable diseases. Sharing integrated data for this purpose ensures a more accurate registry, thus strengthening the services provided to individuals with these conditions.


Investment in statistical and research staff is essential, as they provide the analytical ability and thought processes that empower an integrated data system. Knowledge of data systems and statistical prowess are benchmark skills necessary to develop and maintain such a system. Sufficient information technology skills are important for support roles, but the integrated data system in South Carolina was built primarily through the efforts of statisticians. The South Carolina team possesses a diverse background that includes former clinicians, human services managers, and teachers, as well as mathematicians and statisticians. The professional experiences of the team collectively enrich the analyses required for South Carolina's complex health and human services problems. South Carolina has trained its own generation of integrated data professionals. The real investment of integrated data systems is found in the people who develop and nurture them.

Return to Contents


Existing administrative systems are data rich, and the workforce, given its technical skill and prowess, is capable of using these systems to generate significant change. The next frontier is deploying a community of managers, researchers, advocates and consumers with a research agenda that:

  • Recognizes that health is not organizationally based nor is it captured in a "silo" data system.
  • Focuses on understanding the tangential as well as the core determinants of wellness.

The tools—including the data, computer systems, software, and integration programs—are ready. Full data integration across the health and human services spectrum needs to be a top priority for policymakers and service providers to improve health care for vulnerable populations.

Current as of September 2003

Return to Contents
Return to Tools for Monitoring the Safety Net


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care