Hospitals Face Challenges Using Electronic Health Records to Generate Clinical Quality Measures
Introduction
The American Hospital Association (AHA) commissioned iParsimony, LLC, to conduct a study to investigate hospital experiences with implementation of Meaningful Use Stage 1 electronic clinical quality measures (eCQMs). The study describes the experience with and impact of eCQM implementation in four hospitals – large and small, urban and non-metropolitan – each of which had significant experience with electronic health records (EHRs) prior to Meaningful Use and uses a different EHR from a different vendor company. The findings described in the case study are derived from interviews conducted with key leaders and operational staff directly involved in the oversight and management of eCQMs. Based on the experiences of the hospitals in this case study, the current approach to automated quality reporting does not yet deliver on the promise of feasibility, validity and reliability of measures or the reduction in reporting burden placed on hospitals. Specific policy changes are needed, starting with Stage 2 Meaningful Use, to redirect the electronic clinical quality reporting requirements to focus on a small set of well-tested measures supported by a mature policy infrastructure that can guide valid and feasible measure development, testing and implementation.

Background
In the past 10 years, there has been an unprecedented expansion in the number and type of quality measures hospitals are required to report for federal programs. The Centers for Medicare & Medicaid Services’ (CMS) Hospital Inpatient Quality Reporting (IQR) and Value-Based Purchasing (VBP) programs are two of the drivers of this development. Currently, hospitals report nearly 90 measures across all hospital quality reporting programs, including hospital inpatient and outpatient reporting. EHR-based automated quality measure reporting has the potential, if implemented correctly, to ease the burden of quality reporting, while increasing access to real-time information to support quality improvement. To do so, the quality measure results generated from EHR data must be based upon information that is feasible to collect in an automated fashion, generate valid and reliable results, and thereby demonstrate a benefit that outweighs the costs. The Medicare and Medicaid EHR Incentive Programs, created in the Health Information Technology for Economic and Clinical Health Act of 2009 (HITECH), authorized the Department of Health and Human Services (HHS) to provide financial incentives to hospitals and eligible professionals for the “meaningful use” of certified EHR technology to improve patient care. The 15 eCQMs selected for hospital reporting in the Meaningful Use program Stage 1 are a subset of quality measures included in the hospital IQR program. Across the 15 quality measures, more than 180 data elements are required. The rapid cycle policy development and EHR certification process included a number of factors that likely have affected the accuracy of the eCQM measure results, such as: ■ existing measures modified for calculation by EHRs without robust testing to determine if all of the data were available in existing EHRs; ■ known errors in the eCQMs and lack of implementation within EHRs to test the feasibility of data collection or validation of results; and ■ lack of mature e-specification development and updating process. For additional information on eCQM selection and development, please see the full eCQM case study report by iParsimony, LLC, on the AHA website. Case Study Sites Each of the organizations visited was well situated for success in eCQM adoption. Each showed a strong commitment to health information technology (IT) as a means to enhance its ability to provide safe, effective, efficient, timely, patientcentered and equitable care. Each was externally recognized by national programs for its level of EHR adoption, with significant efforts preceding the Meaningful Use program by five to 10 years. All exhibited a culture of clinician empowerment with strong leadership pushing for health IT to enable quality improvement. There was a common initial belief among the organizations that their significant EHR efforts should lead to a relatively straightforward process to implement the eCQMs.
1

Common Implementation Process Used Across Sites
While the organizations approached eCQM implementation differently, the eCQM implementation required iterations of various steps, as depicted in Figure 1.

Data capture and workflow redesign Gap analysis Data extraction and eCQM calculation
Figure 1. High-level steps in the eCQM implementation process, illustrating an iterative process where data capture and workflow redesign, data extraction and eCQM reporting are interdependent steps, and are influenced by the validation results.

Validation

Downstream uses of eCQM results

Gap analysis All organizations conducted a gap analysis, comparing measure requirements against data captured in the EHR as part of the clinical workflow (the routine patterns of actions, including documentation clinicians use during patient care). The original expectation was that EHRs collect data through the routine care process and that the eCQM reporting tool would accurately extract and report measurements from the existing data, thereby reducing provider burden. They found, however, that much of the needed data was not captured in the required format. Data capture and workflow redesign The data capture and workflow redesign step was perceived as the most involved and onerous step in the expected eCQM implementation workflow process across all sites, regardless of EHR vendor. A particularly resource-intensive step was to modify their EHRs to support data capture of the measure elements. The modification often included the addition of structured fields in the EHR, such as checkboxes and predefined lists of values, because the automatic extraction of the data for the eCQMs required defining a specific location where the data would be automatically recognized by the EHR in order to be used for eCQM calculation. Due to the inflexibility of the certified eCQM reporting tools, each hospital identified a need for workflow modifications to capture required data in specific locations. Data extraction and eCQM calculation The hospitals and health systems studied expected to rely on the eCQM reporting tools to perform the eCQM calculation. Organizations with integrated EHRs (those that include departmental systems in the same database used to store the data captured during clinical care in the
2

EHR) had fewer challenges than organizations with unique departmental systems, regardless of whether these systems were from the same or different vendors. Organizations with unique departmental systems reported no or limited interoperability among the multiple systems, some of which contain information needed for eCQMs. Hence, duplicative workarounds using manual data entry for data already present in departmental systems have been implemented. Validation The organizations expended significant effort to validate eCQM results derived from their respective eCQM reporting tools, but ultimately were unsuccessful. Validation included technical validation and clinical validation: ■ Technical validation included: 1) verifying that all data required by the eCQM reporting tool could be captured in a discrete format in the EHR; and 2) verifying that the specific locations of the data were visible by the eCQM reporting tool for automated data extraction. ■ Clinical validation included: 1) verifying the extent to which clinicians entered the discrete data used by eCQM reporting tool to accurately represent clinical care; and 2) verifying the validity of the eCQM calculation based on the captured data. Organizations reported that validation was an involved and iterative process. One organization struggled to achieve technical validation of its eCQM results and did not achieve clinical validation. Two hospitals were able to validate their technical ability to capture the necessary data; however, the use of these data fields was inconsistent and they did not achieve clinical validation. One hospital achieved technical validation and did not directly compare the results of the eCQMs and the corresponding chart-abstracted measures from which the eCQMs were derived.

As an ongoing step in the eCQM validation process, three organizations developed a staff-intensive and unsustainable concurrent review process to encourage documentation directly by nurses or order-entry by physicians. Assigned staff were charged with continuously reviewing EHR documentation to identify missing data in the EHR that can impact eCQM

results, making the completeness and accuracy of the data used for eCQM calculation dependent on staff review and manual data input. As a result of the concurrent review process, these organizations became more comfortable that the data the eCQMs required were present in the required EHR fields, but it necessitated more ongoing staff work than expected.

Findings
The case study hospitals and health systems were committed to the implementation of eCQMs as part of their overall quality improvement goals. They expected to use eCQMs and Meaningful Use implementation as key tools to achieve their broader quality goals. Specifically, they expected to: to use certified vendor software to capture required data elements for reporting quality metrics as part of an organization-wide commitment to high-level use of EHRs. their eCQM tools or other mechanisms, to improve care by sharing the data with physicians and other clinicians, and empowering them to continuously improve the efficiency and effectiveness of care.

3. Use the EHR for clinical decision support related to eCQMs. Each hospital planned to incorporate clinical decision support to encourage clinicians to deliver care consistent with guidelines (as captured in the measures). Despite extensive EHR implementation experience, each organization quickly learned that the eCQM implementation would be more challenging than anticipated. The actual eCQM implementation required multiple iterations of workflow redesign, data capture, eCQM calculation and validation. Figure 2 depicts the actual eCQM implementation experience.

1. Generate quality data from the EHR. Each hospital sought 2. Use all of their quality data, whether generated through

Install certified eCQM software with prescribed locations for data extraction to report eCQMs Attempt validation of eCQM results by comparing to results from manual chart abstraction - Significant under-reporting of performance identified. - Usability of EHR/eCOM module questioned. Use of system uncovers problems: - data must be in prescribed location or it is not used for eCQM calculation. - internal systems with needed data (e.g., labs) for eCQM not interoperable with EHR. Iterative modifications tried to fix problems 20% of effort by vendor entailed changes to certified EHR technology 80% of effort by hospital entailed changes to hospital work flow solely to accommodate eCQM data capture

Despite continued “above-and-beyond” efforts by hospitals, eCQMs are still inaccurate
Figure 2. High-level steps in the actual eCQM implementation experience, illustrating an iterative process where data capture and workflow redesign, data extraction and eCQM reporting are interdependent steps, and are influenced by the validation results. 3

➧ ➧ ➧ ➧

The findings from the hospitals and health systems studied highlight many challenges: n Program Design Challenges: The eCQM specifications were difficult to access, complex, contained inaccuracies and were not maintained over time. The Meaningful Use Stage 1 eCQM specifications were difficult to find. Unclear specifications and lack of a policy infrastructure to support hospitals and vendors caused confusion and re-work. n Technology Challenges: The eCQM tools from vendors did not work as expected and could not efficiently generate accurate measure results. Hospitals experienced significant rework to create and revamp clinical workflows to meet eCQM tool requirements. In order to accommodate the measure requirements for documentation of specific clinical information in a discrete format, the organizations had to modify their EHRs to support data capture of the measure required elements, adding structured fields (such as checkboxes and pre-defined lists of values). The additional fields, as well as their placement within the EHR forms and navigation, created disconnects from the clinical workflow, which relied on narrative (freetext) documentation in addition to structured data. Hospitals with departmental systems from multiple vendors reported the absence or limited connectivity and sharing of information that may be essential to accurate reporting of the eCQMs. Each organization reports that its vendor is barely staying a step ahead of its customers, given rushed timelines and immature specifications. There is insufficient opportunity to test and develop usable systems with careful customer workflow analysis and to develop smarter ways to capture data given the time constraints of the Meaningful Use program. n Clinical Challenges: The eCQM implementation process negatively affected clinicians, adding to their workload with no perceived benefit to patient care, as it duplicated information already entered in narrative text. Up to two-thirds of the necessary information for eCQM reporting had to be specifically identified in the documentation process in order to capture it for eCQM reporting. In an attempt to capture the needed data, physician and nursing leaders spent considerable time analyzing and making iterative adjustments to clinical workflow so as to not diminish the safety, quality and speed of patient care. The time involved ranged from two to 18 months per measure. Concurrent review was an eCQM validation process used to ensure that data not automatically entered as structured data in the EHR were included as such in EHR screens that the eCQM reporting tool was designed to recognize. As a result, the accuracy of the data used for eCQM calculation is dependent on staff review of entries in the EHR and manual data input. The eCQM-reported measures varied substantially from the chart-abstracted measures from which they were derived. Organizations either spent considerable time in re-work to revise and validate the eCQM measurement process with the eCQM reporting tool, or chose to ignore the results in favor of those derived from the chart-abstracted versions of the measures. n Strategic Challenges: Hospitals expended excessive effort on the eCQMs that negatively affected other strategic priorities. Resources consumed by eCQM implementation delayed other important initiatives, including improving patient engagement and shared decision-making, coordination of care within their communities and regions, streamlining of EHR nursing documentation, and automating the documentation of medications administered using bar codes.

4

Table 1 summarizes the findings and policy recommendations that flow from them. The next section expands on the policy recommendations.

Hospital Experience with eCQM Implementation and Resulting Policy Recommendations

Program Design Challenges eCQMs were introduced before robust testing for validity, accuracy and feasibility. Specifications were hard to find, lengthy and frequently modified to correct errors. Meaningful use eCQMs require unfamiliar vocabularies for data elements (such as LOINC®, SNOMED-CT).

Provider Implications and Adaptive Workarounds n Modifications led to multiple iterations of tools and associated workflow redesign. n Measure results were frequently inaccurate. n Costs to implement were much higher than expected. n Hospitals spent excessive time searching for correct versions or used specifications for chart-abstract measures. n These problems contributed to inaccurate measure results. n Hospitals struggled with unfamiliar vocabularies. n Hospitals relied on eCQM reporting tools to manage the crosswalks between new vocabularies in the eCQMs and the terms used locally or purchased another vendor’s service to support new vocabularies. n Hospitals incurred additional costs. n Hospitals voiced concerns about potential errors in coding or billing and associated risks of subsequent audits. n Hospitals and clinicians saw no benefit from generating inaccurate data. n Hospitals were worried that reporting data that they did not consider to be accurate would create a compliance issue.

Policy Recommendations n Reduce pace of rollout with fewer, but more well-tested measures. n Provide clear guidance and a consistent, reliable process for eCQM development, availability, updating and implementation. n Support the development of an accurate, complete and validated crosswalk from SNOMED-CT to ICD-10-CM and ICD-10-PCS. n Provide for adequate training and education. n Create an eCQM development, testing, and certification program that supports accurate measurement.

Sub-regulatory guidance to ignore data accuracy conflicts with hospital goals for both quality improvement and other program policy to report accurate quality data.

Technology Challenges
EHRs are not designed to capture and enable re-use of information captured during the course of care for later eCQM reporting. EHRs are not designed to capture information from other department information systems at the level of detail needed for eCQM reporting. EHR vendors update and separately deliver individual EHR components for Meaningful Use.

Provider Implications and Adaptive Workarounds n Hospital clinical staff enter information multiple places in EHRs to ensure data availability for eCQM reporting. n Staff time devoted to manual re-entry of information that already exists elsewhere in the EHR reverses efficiencies gained from the use of EHRs and undermines the presumed value of automation for quality reporting and improvement. n Quality or other staff abstract information from other department information systems and enter it into the fields in the EHR required to report the eCQMs.

Policy Recommendations n Improve heath IT standards for EHRs and eCQM reporting tools to address usability and data management. n Improve vendor tools to include workflow design flexibility. n Improve EHRs and reporting tools to support intra-hospital interoperability.

n Hospitals conducted multiple updates and iterative testing.

n Establish a predictable update process and schedule for eCQMs with easy access and notification of updates. n Require vendors to support the latest update on a specified schedule.
5

Clinical Challenges
EHRs and certification requirements are not designed to support effective and efficient patient care workflows or draw data from them. Hospitals were unable to validate the eCQM results.

Provider Implications and Adaptive Workarounds n Hospitals modified workflows solely to support adequate data capture, working iteratively with their vendors. n Ultimately, hospitals substantively altered clinical workflow solely to accommodate the data needed for the eCQMs, with no benefit for patient care. n Hospitals either reported the results of eCQMs as inaccurate, but a work in progress, or did not report the eCQM results directly to physicians and nurses. n Inaccurate results from the eCQM reporting tool combined with increased workflow requirements led to clinicians mistrusting the data and not using it for care improvement. n Physicians who use up-to-date sets of orders may cause the hospital to have poorer performance as measured by the eCQMs.

Policy Recommendations n Give vendors more time to develop useful and accurate tools that support logical workflows and leverage data already in the EHR. n Create an eCQM development, testing, and certification program that supports accurate measurement.

Meaningful use Stage 1 eCQM specifications are out-of-date and sometimes inconsistent with current care recommendations.

n Create a mechanism to update eCQMs to reflect new state-ofthe-art clinical practice and to match updates to corresponding chart abstracted measures.

Strategic Challenges
Time and personnel requirements to implement eCQMs were far beyond expectations and excessive.

Provider Implications and Adaptive Workarounds n Hospitals added tasks to existing IT and/or quality management staff responsibilities and delayed projects. n Clinical staff expended considerable time documenting for eCQMs, with no perceived value for patient care. n Excessive staff time spent on eCQMs delayed focus on other priorities such as reducing readmissions, improving patient safety or advancing care coordination. n Results damaged credibility of hospital leadership and Meaningful Use program as a whole.

Policy Recommendations n Consider the effort required in future policy for eCQMs.

Combination of time and effort involved and inability to validate results meant hospitals saw no return on investment.

n Reduce pace of rollout with fewer, but more well-tested measures that can be generated by tools that support logical workflows and leverage data already in EHRs.

6

Policy Recommendations
Many hospitals, including the four facilities studied for this report, have shown a commitment to using their EHRs and eCQMs as a method to automate the quality measurement process. Hospitals have undertaken this task with the belief that the technology would support the development of automated clinical quality reporting and assist their local quality improvement initiatives. However, the most advanced hospitals have been unable to use their EHRs to efficiently implement the eCQMs and achieve accurate results. Based on the experience of these four hospitals, specific policy changes are needed, starting with Stage 2 Meaningful Use to redirect the electronic clinical quality reporting requirements to focus on a small set of well-tested measures supported by a mature policy infrastructure that can guide valid and feasible measure development, testing and implementation:

2. Make EHRs and eCQM reporting tools more flexible 3. Improve health IT standards for EHRs and eCQM so that data capture can be aligned with workflow and interoperable so that data can be shared across hospital department systems. reporting tools to address usability and data management to achieve Meaningful Use program expectations. Additional EHR fields to capture structured clinical information disrupts the usual clinical workflow, increases time on narrative and structured data documentation and decreases time for patient care. adopting them in national programs. Implement eCQMs within hospitals as part of testing to ensure information flow is accurate and there is no adverse impact on quality and patient safety.

4. Carefully test eCQMs for reliability and validity before

1. Slow the pace of the transition to electronic quality

reporting with fewer but better-tested measures, starting with Stage 2. The additional time would allow: n policymakers to create a reliable policy process for eCQM implementation, a mechanism to provide eCQM updates, and a robust EHR testing/certification program; n vendors to develop tools that support logical workflows, produce accurate measures and leverage all data already in the EHR; and n hospitals to implement the tools in a way that supports their quality goals without excessive burden or risk to patients.

5. Provide clear guidance and tested tools to support successful hospital transition to increased electronic quality reporting requirements. For example, develop and disseminate an accurate, complete and validated crosswalk from SNOMED-CT®, a vocabulary required by the 2014 eCQMs and Meaningful Use Stage 2, to ICD-10-CM for conditions and ICD-10-PCS for procedures.

Conclusion
This study demonstrates that successful implementation of current policy requirements for eCQMs must be redirected so that the EHRs are working for the clinicians rather than the clinicians spending extensive amounts of time working for the EHRs. The current process raises costs and effort for providers without leading to accurate data and is not sustainable. More time is needed to develop and test a smaller set of eCQMs and demonstrate their feasibility, reliability, validity and usability. Unless eCQMs are more robustly tested before incorporation in widespread national programs such as Meaningful Use, the reality is that hospitals will not be able to meet the new standards and certified EHRs will not deliver on the promise of eCQMs to improve patient care, provide comparative quality information to consumers and reduce the burden of quality reporting.

i

HITECH Act, title XIII of division A and title IV of division B of the Recovery Act. Pub. L. No. 111-5, div. A, tit. XIII, 123 Stat. 115, 226-279 and div. B, tit. IV, 123 Stat. 115, 467-496 (2009) 7

325 Seventh Street, NW Suite 700 Washington, DC 20004-2802 202.638.1100 www.aha.org
©2013 American Hospital Association

7/24/13 17:00