An analysis of publicly reported pediatric heart surgery data and patient mortality implications

Introduction

Since the 1990s, a consistent body of evidence has identified that higher hospital volumes of a complex surgery are associated with better patient outcomes, including lower mortality rates, lower complication rates, and shorter lengths of stay (1-5). This volume-outcome relationship has been identified for many different surgical therapies, including cancer, cardiac, gynecological, head and neck, and orthopedic, as well as some medical conditions. The relationship has been previously identified for pediatric and congenital heart surgery in the USA and internationally (6-10), and is generally strongest for more complex and perhaps less common procedures (2,11). It is also important to acknowledge that although a volume-outcome relationship may exist, this is based on an analysis of aggregate data, and that some low-volume programs obtain excellent outcomes.

Since 1994, the Society of Thoracic Surgeons (STS) has included data on congenital and pediatric cardiothoracic patients in its clinical registry. More than 95% of hospitals that perform pediatric cardiac surgery in the USA participate in the STS Congenital Heart Surgery Database (CHSD), and the patient-level penetration is even higher, because virtually all high-volume pediatric cardiac surgical programs participate in the STS CHSD (12). One of the data points captured in the registry is operative mortality, defined by STS as (I) all deaths, regardless of cause, occurring during the hospitalization in which the operation was performed, even if after 30 days (including patients transferred to other acute care facilities); and (II) all deaths, regardless of cause, occurring after discharge from the hospital, but before the end of the 30^th postoperative day (13,14). The STS registry stratifies pediatric and congenital heart operations into five categories based on the risk of in-hospital mortality [i.e., The Society of Thoracic Surgeons-European Association for Cardio-Thoracic Surgery (STAT) Mortality Categories 1-5]. STAT Category 1 operations are those associated with the lowest risk for mortality, and STAT Category 5 operations are those associated with the highest risk for mortality (15).

In 2014, STS developed a comprehensive STS CHSD mortality risk model which comprehensively adjusts for case mix and comorbidities by taking into consideration patient-level data not limited to the procedure performed (16,17). It was the development, validation, and implementation of this risk model that made it possible in 2015 for STS to offer voluntary public reporting on their website of center-level performance for pediatric and congenital cardiac surgery. As of February 2017, 61 of 112 (54%) of hospitals in the USA that participate in the STS CHSD had chosen to publicly report their pediatric and congenital heart surgery data. When publicly reporting hospital data, STS reports information about all procedures as well as data about procedures within each STAT category. These data include the number of operations performed, the number of observed deaths, the overall operative mortality rate, the expected mortality rate based on the STS risk model, the observed-to-expected (O/E) operative mortality ratio, and the overall adjusted mortality rate (15).

A “volume-outcome” relationship suggests that patient deaths could potentially be avoided if operations performed in low-volume hospitals could be shifted to high-volume hospitals, especially for the most complex operations. The potential magnitude of this phenomenon in pediatric and congenital heart surgery patients has yet to be described. Using the publicly available STS registry data, the goal of this study was to understand the number of deaths of pediatric and congenital heart surgery patients that theoretically could be avoided if the highest-risk patients all had surgery at high-volume hospitals.

Methods

We downloaded data in February 2017 from The Society of Thoracic Surgeons website for the 61 hospitals that choose to publicly report their outcomes. These included operations performed from January 2012 to December 2015 and were the most recent publicly available data. Given that the reported data were aggregated at a hospital-level and were already in the public sphere, approval by our university’s institutional review board was not required for this study.

We categorized each hospital in the data set as high-, medium-, or low-volume, using STS’s previously identified category thresholds based on the hospital’s mean annual volume of pediatric and congenital heart surgery. Low-volume hospitals were those that averaged fewer than 100 index cardiac operations per year; medium-volume hospitals were those that averaged 100 to 249 index operations per year; and high-volume hospitals were those that averaged 250 or more index operations per year. For purposes of calculating the potential mortality impact, we limited our analysis to STAT mortality 4 and 5 category operations, as previous research had identified the strongest volume-outcome relationship in these higher STAT categories (6,18,19).

Using the expected mortality rates calculated by STS for STAT category 4 and 5 operations at each hospital (16,17), we multiplied the expected mortality rate by the number of procedures performed in that STAT category at that hospital to calculate the expected number of deaths for that STAT category at that hospital. For each volume category (high-, medium-, and low-volume) and STAT category (STAT mortality 4 or 5) combination among hospitals, we totaled the number of procedures performed, the number of observed deaths, and the number of expected deaths. Then for each volume category and STAT category combination, we calculated an observed mortality rate, an expected mortality rate, and an observed-to-expected (O/E) ratio.

As high-volume hospitals had the lowest O/E ratio, these were considered the reference in terms of mortality performance. We calculated the number of deaths that theoretically would have occurred in medium- and low-volume hospitals if they performed at the same level (O/E ratio) as high-volume hospitals. We then subtracted this number of deaths from the actual number of deaths in the low- and medium-volume hospitals to estimate the number of deaths that could potentially have been prevented. We translated the number of deaths prevented into a percentage reduction by dividing the number of deaths prevented by the number of observed deaths. Finally, we calculated 95% confidence intervals (CI) for our projected reductions to quantify the statistical uncertainty in our estimates.

Our analysis included the data for the 61 hospitals that choose to have their STS data reported publicly and did not make any adjustments for hospitals that choose not to report their data publicly. The quality of the data in the STS CHSD is evaluated through intrinsic data verification and audit of participating sites. Audits have confirmed that the data in the STS CHSD are complete and accurate.

Results

Table 1 reflects the findings from our analysis. Of the 61 hospitals for which data were available, 26 hospitals were categorized as high-volume, 22 hospitals were categorized as medium-volume, and 13 hospitals were categorized as low-volume. For the STAT mortality 4 and 5 category operations, high-volume hospitals had an aggregate observed mortality rate of 7.1%, versus an expected rate of 8.4%; medium-volume hospitals had an aggregate observed mortality rate of 9.5%, versus an expected rate of 8.4%; and low-volume hospitals had an aggregate observed mortality rate of 9.9%, versus an expected rate of 8.7%. A similar pattern in observed and expected mortality rates was found when analyzing STAT mortality 4 and 5 operations separately.

The O/E ratios (0.8, 0.9) for STAT 4 and 5 category pediatric and congenital operations at high-volume hospitals were better in aggregate than the aggregate O/E ratios for medium-volume hospitals (1.2, 1.0) and low-volume hospitals (1.4, 1.1) for these same operations. We calculated that 84.5 and 19.5 deaths in STAT 4 and STAT 5 category operations, respectively, theoretically might have been prevented over the 4-year data reporting period if the STAT 4 and 5 category operations done at low- and medium-volume hospitals were performed with the same outcome performance as high-volume hospitals, reflecting an overall 26% reduction in observed deaths.

Hospitals that choose to publicly report their STS data are more likely to be a medium- or large-volume hospital and to be assigned a two- or three-star rating from STS (Table 2). STS assigns a star rating for each hospital based on their overall O/E risk adjusted operative morality ratio—one star reflects a hospital that has higher than expected operative mortality (the 95% CI for their risk adjusted O/E mortality ratio was entirely above the number 1), two stars reflects a hospital that has same as expected operative mortality (the 95% CI for their risk adjusted O/E mortality ratio overlapped with the number 1), and three stars reflects a hospital that has lower than expected operative mortality (the 95% CI for their risk adjusted O/E mortality ratio was entirely below the number 1). The likelihood of a hospital choosing to publicly report their STS data increases as the hospital’s volume category increases and their star rating increases.

Discussion

In our analysis of 61 hospitals that choose to publicly report their STS pediatric and congenital heart surgery performance data on STS’s website, we identified that pediatric cardiac surgery patients who had the most complex operations (STAT 4 or 5 operations) could potentially experience a 26% relative reduction in mortality if their operations were done at high-volume hospitals instead of low- or medium-volume hospitals. Among the hospitals in our database, this would result in preventing 26 deaths per year. These findings represent the lower end of the estimate of potential deaths that could be prevented, as we did not extrapolate our results to STS participating hospitals that choose not to publicly report their data, nor to those few hospitals that perform pediatric and congenital heart surgery but do not participate in STS.

Our findings were very consistent with those of Chang and Klitzner in their study of pediatric and congenital heart surgery patients in 20 California hospitals between 1995–1997; these investigators estimated a 24% relative mortality reduction that theoretically might have occurred if all patients were referred from low- and medium-volume hospitals to high-volume, high-performing institutions (20). There were several minor differences between their study and ours. The Chang study included all surgical cases, regardless of complexity, while our study focused on the higher-risk STAT category 4 and 5 operations, which are performed in smaller numbers. Additionally, their study looked specifically at high-volume, high-performing hospitals, while our study investigated high-volume hospitals as a group and made no distinction between whether they were high- or low-performing. Nevertheless, our study is consistent with studies published 20 years ago that reveal that many low-volume hospitals have higher mortality than high-volume hospitals for high complexity procedures.

Our analysis required a number of assumptions and had some limitations. First, the cut points for the volume categories that have been identified by STS apply to the total number of pediatric and congenital heart surgery procedures performed at the hospital; they do not account for the distribution of cases across the five STAT mortality risk categories. For example, a high-volume hospital could possibly perform large numbers of low-risk operations and very few high-risk operations, which would not necessarily be differentiated with the current volume categorizations. We assumed that the experience with all procedures was reflective of the hospital’s experience with the highest-risk procedures (STAT category 4 and 5 operations). Different volume category cut points could have been used for the analysis, but given that STS had already defined the volume categories, we thought it would be best to use an established standard, rather than arbitrarily defining new cut points ourselves.

Second, we used the expected mortality rates provided by STS, which were based on a risk-adjustment model developed by STS: the STS CHSD mortality risk model (16,17). This risk-adjustment model has good model fit and discrimination with overall C statistics of 0.875 and 0.858 in the development sample and the validation sample, respectively. These C statistics are the highest C statistics ever seen in a pediatric cardiac surgical risk model. As the STS CHSD mortality risk model is the best available model to date for measuring outcomes after pediatric cardiac surgery and has been endorsed by the National Quality Forum (15), we felt safe assuming that the STS risk model captures all of the appropriate patient-level risk factors.

Third, given that the STS public website only reports facility-level data, not patient-level data, and that the STAT mortality risk categories were the only indication of ‘risk’ provided in the public data, we had to assume that patients within a STAT mortality risk category carried the same risk across low-, medium-, and high-volume hospitals.

Finally, we recognize that our findings are somewhat sensitive to the hospitals that choose to publicly report their data. Our results may be different if a different set of hospitals chose to publicly report their data. But given that all three-star hospitals chose to publicly report their data and most high-volume hospitals chose to publicly report their data, and the smaller volume of surgery performed in the lower-volume hospitals, we would not anticipate our results to shift drastically with reasonable changes in reporting hospitals. If data from all STS participating hospitals were made available, we would be able to test this assumption empirically.

Conclusions

This analysis identified a potentially large relative risk reduction in mortality if the highest risk pediatric and congenital heart operations currently being performed at low- and medium-volume hospitals could have been shifted to high-volume hospitals.

A number of possible solutions exist to reduce these potentially preventable deaths. First, hospital performance needs to be made transparent for all hospitals, not just those who voluntarily agree, so that patients and families are aware of all the facts when making a decision about providers. All hospitals that perform pediatric and congenital heart surgery should publicly report their STS data. As it is unlikely that this will occur voluntarily, patient and family organizations, as well as payers and regulators, should insist on universal transparency of pediatric cardiac surgical outcomes data. As a professional society, STS has developed outcomes registries as platforms for quality assessment and quality improvement, and a website for voluntary sharing of this information by programs that choose to participate. However, participation in the STS Congenital Heart Surgery Database is voluntary and STS cannot require hospitals to make their outcomes data public. Sixty-one hospitals that participate in the STS registry have already chosen to make their data transparent, including some low- and medium-volume hospitals. This transparency should be applauded. Other STS-participating hospitals should be encouraged to do the same. While the relationship between hospital volume and patient outcomes has demonstrated a consistent and strong relationship in virtually every complex procedure that has been evaluated, we need to better understand why this relationship exists and how lower-volume hospitals might improve.

Second, STS currently uses a star rating system on its public reporting website, which is based on the hospital’s overall performance on pediatric and congenital heart surgery procedures for all levels of surgical risk (STAT categories 1-5). STS also reports risk-adjusted mortality outcomes for each individual STAT category. Ideally, to facilitate consumer understanding, STS might also report star ratings for individual STAT categories. However, this is statistically challenging because of the lower procedural volumes (i.e., small sample size). One possible compromise would be to report combined outcomes for STAT categories 4 and 5; this approach might provide the public with more easily understood information on those operations for which experience has been shown to be most important.

Finally, given the number of possible preventable deaths that occur every year, the greatest policy issue is for providers and policy makers to explore regionalizing care, not just for the highest risk pediatric and congenital cardiac procedures but for similar high complexity procedures in other specialties as well. Regionalization could reduce operative risks, assure access for all patients with complex conditions to the highest quality care, and provide an optimal educational environment for surgical trainees. These benefits would have to be balanced against the desire of many patients to receive care in their home communities (21), and the cost and logistical challenges of having patients travel to regional centers for complex procedures. Federal and local leaders would have to develop solutions that balance these different goals and meet the needs of their community.

Given patient preferences for receiving care locally when possible, and the strong economic and reputational incentives associated with pediatric cardiac surgery, we recognize that regionalizing care will not be easy and will require committed leadership. One possible place to start with regionalizing care might be in metropolitan areas, where low-volume hospitals could seek to move their volume to geographically proximate medium- and high-volume hospitals, which might reduce preventable deaths while still maintaining patient access and satisfaction.

These policy changes will not be easy; nothing worthwhile ever is. Yet, given the substantial number of potentially preventable deaths related to high complexity pediatric and congenital heart surgery outside of high-volume hospitals, it is morally imperative that surgeons, hospitals, professional societies, and policy makers address this issue.

Acknowledgments

The authors thank Marshall Jacobs, MD, from Johns Hopkins University, Department of Surgery, Paul Kurlansky, MD, from Columbia University, Department of Surgery, and Vinay Badhwar, MD, FACS, FACC, from West Virginia University, Department of Surgery, for their thoughtful review of the paper. The authors also thank Steve Sternberg from US News and World Report for drawing our attention to the publicly available data set and for his thoughtful assistance in helping us conceptualize the analysis.

Funding: None.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jhmhp.2018.11.01). JM Austin discloses grant or contract support from The Leapfrog Group and the Agency for Healthcare Research and Quality (for research related to hospital performance measurement). JM Derk discloses contract support from The Leapfrog Group (for research related to hospital performance measurement). PJ Pronovost discloses grant or contract support from the Agency for Healthcare Research and Quality, and the Gordon and Betty Moore Foundation (research related to patient safety and quality of care), the National Institutes of Health (acute lung injury research), the American Medical Association (improve blood pressure control), and various health systems (implement safety programs). He has received honoraria from various hospitals, health systems, and the Leigh Bureau to speak on quality and patient safety. He has board membership to the Cantel Medical Group and is founder of Doctella, a patient engagement platform.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Birkmeyer JD, Siewers AE, Finlayson EV, et al. Hospital volume and surgical mortality in the United States. N Engl J Med 2002;346:1128-37. [Crossref] [PubMed]
2. Reames BN, Ghaferi AA, Birkmeyer JD, et al. Hospital volume and operative mortality in the modern era. Ann Surg 2014;260:244-51. [Crossref] [PubMed]
3. Dudley RA, Johansen KL, Brand R, et al. Selective referral to high-volume hospitals: Estimating potentially avoidable deaths. JAMA 2000;283:1159-66. [Crossref] [PubMed]
4. Begg CB, Cramer LD, Hoskins WJ, et al. Impact of hospital volume on operative mortality for major cancer surgery. JAMA 1998;280:1747-51. [Crossref] [PubMed]
5. Shahian DM, Normand SL. The volume-outcome relationship: From Luft to Leapfrog. Ann Thorac Surg 2003;75:1048-58. [Crossref] [PubMed]
6. Welke KF, O'Brien SM, Peterson ED, et al. The complex relationship between pediatric cardiac surgical case volumes and mortality rates in a national clinical database. J Thorac Cardiovasc Surg 2009;137:1133-40. [Crossref] [PubMed]
7. Jenkins KJ, Newburger JW, Lock JE, et al. In-hospital mortality for surgical repair of congenital heart defects: Preliminary observations of variation by hospital caseload. Pediatrics 1995;95:323-30. [PubMed]
8. Hannan EL, Racz M, Kavey RE, et al. Pediatric cardiac surgery: The effect of hospital and surgeon volume on in-hospital mortality. Pediatrics 1998;101:963-9. [Crossref] [PubMed]
9. Lundstrom NR, Berggren H, Bjorkhem G, et al. Centralization of pediatric heart surgery in Sweden. Pediatr Cardiol 2000;21:353-7. [Crossref] [PubMed]
10. Kansy A, Ebels T, Schreiber C, et al. Higher programmatic volume in paediatric heart surgery is associated with better early outcomes. Cardiol Young 2015;25:1572-8. [Crossref] [PubMed]
11. Finks JF, Osborne NH, Birkmeyer JD. Trends in hospital volume and operative mortality for high-risk surgery. N Engl J Med 2011;364:2128-37. [Crossref] [PubMed]
12. Jacobs JP, Mayer JE Jr, Mavroudis C, et al. The Society of Thoracic Surgeons Congenital Heart Surgery Database: 2017 update on outcomes and quality. Ann Thorac Surg 2017;103:699-709. [Crossref] [PubMed]
13. Jacobs JP, Mavroudis C, Jacobs ML, et al. What is operative mortality? Defining death in a surgical registry database: A report of the STS Congenital Database Taskforce and the Joint EACTS-STS Congenital Database Committee. Ann Thorac Surg 2006;81:1937-41. [Crossref] [PubMed]
14. Jacobs JP, O'Brien SM, Pasquali SK, et al. The importance of patient-specific preoperative factors: An analysis of the Society of Thoracic Surgeons Congenital Heart Surgery Database. Ann Thorac Surg 2014;98:1653-8; discussion 1658-9. [Crossref] [PubMed]
15. Jacobs JP. The Society of Thoracic Surgeons Congenital Heart Surgery Database public reporting initiative. Semin Thorac Cardiovasc Surg Pediatr Card Surg Annu 2017;20:43-8. [Crossref] [PubMed]
16. O'Brien SM, Jacobs JP, Pasquali SK, et al. The Society of Thoracic Surgeons Congenital Heart Surgery Database mortality risk model: Part 1-statistical methodology. Ann Thorac Surg 2015;100:1054-62. [Crossref] [PubMed]
17. Jacobs JP, O'Brien SM, Pasquali SK, et al. The Society of Thoracic Surgeons Congenital Heart Surgery Database mortality risk model: Part 2-clinical application. Ann Thorac Surg 2015;100:1063-8; discussion 1068-70. [Crossref] [PubMed]
18. Pasquali SK, Li JS, Burstein DS, et al. Association of center volume with mortality and complications in pediatric heart surgery. Pediatrics 2012;129:e370-6. [Crossref] [PubMed]
19. Danton MH. Larger centers produce better outcomes in pediatric cardiac surgery: Regionalization is a superior model - the con prospective. Semin Thorac Cardiovasc Surg Pediatr Card Surg Annu 2016;19:14-24. [Crossref] [PubMed]
20. Chang RK, Klitzner TS. Can regionalization decrease the number of deaths for children who undergo cardiac surgery? A theoretical analysis. Pediatrics 2002;109:173-81. [Crossref] [PubMed]
21. Finlayson SR, Birkmeyer JD, Tosteson AN, et al. Patient preferences for location of care: Implications for regionalization. Med Care 1999;37:204-9. [Crossref] [PubMed]