To risk adjust or not to risk adjust for sociodemographics—that is the question, so what is the answer?

It is estimated that 606,880 people died from cancer in the United States in 2019 (1). Three of the most commonly diagnosed malignancies include breast, colon, and lung cancer which have well defined treatment modalities that often includes the use of chemotherapy. The National Quality Forum (NQF) and the Commission on Cancer (CoC) of the American College of Surgeons have developed and endorsed several guidelines and metrics involving the rate of appropriate administration of chemotherapy (2,3) among eligible cancer patients. Hospitals are rated and accredited based on thresholds of compliance for these chemotherapy metrics and which are used as a measure of hospital quality. Thus, understanding how these metrics are calculated and used in practice has important ramifications for benchmarking cancer care quality at the hospital-level.

Currently, the NQF and the CoC measures consider a hospital compliant if chemotherapy was recommended to a patient but they did not receive it because they either refused or were lost to follow-up. In their article (4), Ellis et al. argue that while this appropriately does not penalize hospitals for not offering appropriate chemotherapy, it may mask a true gap in cancer care quality that may be associated with socioeconomic disparities. Whether to adjust for and incorporate measures of socioeconomic factors in quality measures has been a controversial policy topic. In particular, it has been argued that social risk adjustment may obscure true performance by not penalizing providers who deliver low quality care to disadvantaged populations, whereas others argue that it may inappropriately penalize providers who work for safety net hospitals with a large proportion of socially disadvantaged patients (5,6). To this end, the objectives of the study conducted by Ellis et al. were to identify patient-level factors associated with failure to receive recommended chemotherapy and to characterize hospital-level variation in failure to administer chemotherapy without a documented contraindication separately for breast, colon, and lung cancer patients.

The study identified patients diagnosed with colon, breast, and lung cancer from 2005 to 2015 in the United States using the National Cancer Database (NCDB), a hospital-based registry that capture approximately 70% of all newly diagnosed cancers. In total, 183,148 patients treated at 1,281 hospitals met inclusion criteria for the study. The rate of failure to receive chemotherapy for breast (n=82,598), colon (n=90,077), and lung cancer (n=8,473) was 3.5%, 6.6%, and 10.7%, respectively. By cancer type, investigators used multivariable logistic regression models accounting for hospital clustering in order to identify independent patient-level characteristics associated with failure to receive chemotherapy. In general, socioeconomic disparities were reported. For example, non-Hispanic black compared to white breast and colon cancer patients had 26% and 38% increased odds of failing to receive chemotherapy, respectively. Insurance status also predicted failure to receive chemotherapy. For example, among lung cancer patients, those who were uninsured or had Medicaid coverage had 40% increased odds of failing to receive chemotherapy compared to those with Medicare insurance. Furthermore, differences by median household income in Census zip code were observed for breast and colon cancer patients.

In subsequent hospital-level analyses, Ellis et al. separately estimated hospital-level chemotherapy compliance rates using the existing CoC compliance definition and using the updated definition which excludes patients failing to receive recommended chemotherapy from the numerator. This allowed them to estimate unadjusted, hospital-specific rates of failure to administer recommended chemotherapy. While the overall rates were low, they observed wide variation in hospital-specific rates of failure to administer recommended chemotherapy allowing them to identify high and low performers. For example, the maximum rates of hospital-level failure to administer physician recommended chemotherapy were 21.8% in breast cancer, 40.2% in colon cancer, and 40.0% in lung cancer. Furthermore, using unadjusted hierarchical logistic regression models, investigators estimated hospital-level odds ratios and corresponding 95% confidence intervals and identified 95 hospital high outliers (poor performers) for breast cancer, 184 in colon cancer, and 18 in lung cancer. However, when models were adjusted for sociodemographics, 44 hospitals that were identified as high outliers in unadjusted analyses were re-categorized as hospitals with average rates.

The results of this study must be interpreted in light of several important limitations. First, the quality measures used in the study were developed at different time points, and it is unclear to what extent uptake of knowledge of the measures varied by hospital. Second, while the NCDB documents patient refusal to receive chemotherapy, lack of detailed data regarding the conversation and reasons why they refused chemotherapy would be important to measure in future studies. Third, it would be important to measure more detailed patient-level sociodemographic data. For example, this study used Census derived median household income and education data at the zip code level, which may not be a true measurement of patient-level socioeconomic status. Finally, cancer quality of care may vary by providers within hospitals, thus understanding which provider-level characteristics are associated with compliance in recommended guidelines.

Despite these limitations, the results from this study have important ramifications for national stakeholders including accreditation organizations, physician groups, policy makers, and healthcare organizations. Since the Institute of Medicine published its influential report of “Ensuring Quality Cancer Care” in 1999 outlining the complexities and deficiencies in cancer care, there has been a growing interest in fulfilling its goal to develop an ideal cancer care system that is effective, safe, and equitable. In particular, multiple payment and policy reforms in the US during the last two decades have shifted the focus of healthcare from volume to value, and have increasingly coupled provider reimbursement to process, outcome, and performance metrics in an effort to reduce healthcare costs and increase delivery of high-quality care (7). This has made the measurement and estimation of quality measures an important undertaking for policy makers and researchers. However, there has been extensive debate and disagreement regarding whether social factors should be accounted for and adjusted for in process measures (8). With respect to chemotherapy process measures, this discrepancy may be reflected in the fact that current CoC measures consider a patient who did not receive chemotherapy but was recommended to by the physician compliant on the measure. There is concern that sociodemographic variables may be associated with refusal to receive chemotherapy and that patients may “slip through the cracks” of the healthcare system.

The most important finding of this study was that sociodemographic characteristics (including race/ethnicity, type of insurance, and median income) were associated with rates of failure to administer recommended chemotherapy. Furthermore, these rates varied widely by hospital whereby some hospitals were almost always compliant, and some hospitals had as many as 40% of their patients not receiving recommended chemotherapy. When incorporating sociodemographic variables in identifying poor performers, 95 hospitals that were identified as poor performers in unadjusted analyses were no longer poor performers suggesting that these hospitals were being inappropriately penalized for having a large percentage of disadvantaged patients. Thus, Ellis et al. concluded that hospital-specific rates of failure to administer recommended chemotherapy are measures of hospital quality, but are currently not being incorporated into measure definitions, thus causing a “blind spot” whereby some hospitals’ performance is being mischaracterized. They argue that current quality measure definitions should be reconsidered, and that they should be modified in order to limit a hospital’s ability to be rewarded for cases where chemotherapy was recommended but not ultimately delivered. This study contributes to the greater conversation regarding the development and use of quality measures as a way to benchmark the delivery of care in the US in an effort to improve patient outcomes. Specifically, the results from this study corroborates research evaluating the challenges and implications of social risk adjustment in a variety of healthcare settings. For example, one study reported that among practices participating in Medicare’s Value Based Payment Modifier program, risk adjustment of social factors considerably affected estimation of performance measures between practices serving high versus low risk patients suggesting that Medicare’s pay for performance program has exacerbated, not reduced disparities (9). Furthermore, another study reported that physician participation in Accountable Care Organizations (ACOs) varies dramatically by sociodemographic characteristics and was lowest among areas that served a large percentage of black, impoverished, and uninsured patients (10). This suggests that current patterns of physician participation in advanced payment models such as ACOs are associated with receipt of lower quality of care among vulnerable populations. These are just two examples of unintended consequences of recent delivery system innovations that may be associated with shortcomings in providing optimal and equitable care for patients of lower socioeconomic status.

In conclusion, using NCDB data among breast, colon, and lung cancer patients, Ellis et al. identified several sociodemographic disparities associated with failure to receive recommended chemotherapy and noted that hospital-level rates of this measure varied widely by hospitals. They argue that the current quality measure definitions endorsed by NQF and the CoC are inadequate to capture hospital-level performance in administering chemotherapy because they do not appropriately adjust for social risk factors that may explain reasons for not adhering to quality measures. The answer to the question posed by the title of their study is that failure to administer recommended chemotherapy is indeed a cancer care quality blind spot and should not be interpreted as acceptable variation. Research investigating how to appropriately adjust for social risk factors when measuring physician and hospital performance is warranted. Future policies and strategies should continue to investigate ways to provide optimal, high-quality cancer care, especially for vulnerable populations.

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Journal of Hospital Management and Health Policy. The article did not undergo external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jhmhp.2020.03.01). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. National Cancer Institute. Cancer stat facts: cancer of any site. Available online: https://seer.cancer.gov/statfacts/html/all.html
2. Hewitt M, Simone JV. editors. Ensuring quality cancer care. Washington, DC: National Academy Press, 1999.
3. American College of Surgeons. CoC quality of care measures. Expected performance rates for 2020 surveys are based on 2016 standards 4.4 and 4.5. 2019. Available online: https://www.facs.org/quality-programs/cancer/ncdb/qualitymeasures
4. Ellis RJ, Schlick CJR, Feinglass J, et al. Failure to administer recommended chemotherapy: acceptable variation or cancer care quality blind spot? BMJ Qual Saf 2020;29:103-12. [Crossref] [PubMed]
5. Chien AT, Chin MH, Davis AM, et al. Pay for performance, public reporting, and racial disparities in health care: how are programs being designed? Med Care Res Rev 2007;64:283S-304S. [Crossref] [PubMed]
6. National Quality Forum. Risk adjustment for socioeconomic status or other sociodemographic factors. Washington, DC: National Quality Forum, 2014.
7. Miller HD. From volume to value: better ways to pay for health care. Health Aff (Millwood) 2009;28:1418-28. [Crossref] [PubMed]
8. Fiscella K, Burstin HR, Nerenz DR. Quality measures and sociodemographic risk factors: to adjust or not to adjust. JAMA 2014;312:2615-6. [Crossref] [PubMed]
9. Roberts ET, Zaslavsky AM, McWilliams JM. The Value-Based Payment Modifier: Program Outcomes and Implications for Disparities. Ann Intern Med 2018;168:255-65. [Crossref] [PubMed]
10. Yasaitis LC, Pajerowski W, Polsky D, et al. Physicians' participation in ACOs is lower in places with vulnerable populations than in more affluent communities. Health Aff (Millwood) 2016;35:1382-90. [Crossref] [PubMed]