CMS program participation and policy evaluation without administrative data: a case study on Bundled Payments for Care Improvement (BPCI) initiative
Introduction
The US spends significantly more per capita on healthcare than other developed countries. Some reasons for this include lack of health insurance coverage, increased hospital fees due to recent hospital consolidation, high drug prices, and wasteful spending. The Centers for Medicare and Medicaid Services (CMS) has attempted to solve this problem by increasing the value of US healthcare through improvements in health outcomes and/or decreases in health-related costs. Specifically, CMS has created various alternative payment models (APMs) that use financial incentives to reward providers for delivering higher value care, including population-based shared savings programs, patient centered medical home models, and bundled payments.
Recently, CMS announced a greater emphasis on physician leadership in APMs (1). In 2013, CMS expanded its bundled payment APM through the Bundled Payments for Care Improvement (BPCI) Initiative for participating acute care hospitals and physician group practices (PGPs). BPCI builds on Medicare’s prospective payment system by paying a lump sum to providers for not only the acute care hospital stay, but also physician payments and all other spending up to 90 days after a hospitalization to encourage providers to coordinate their services and be more cost-conscious. Bundled payments in theory increase value by decreasing unnecessary healthcare costs, improving quality of care, and improving patient outcomes.
In 2018, CMS continued expanding its bundled payment APM through the BPCI Advanced Initiative (2). Although the impact of BPCI on participating hospitals has been evaluated, an understanding of the impact on PGPs has lagged, in large part due to an initial lack of available CMS administrative lists of participating physicians (3). To assess whether health policy researchers could adequately evaluate the impact of BPCI on PGP physicians without CMS administrative data, we investigated the accuracy of using data from available non-CMS sources to identify BPCI physicians.
Methods
National Provider Identifiers (NPIs) for individual physicians participating in phase 2 of BPCI model 2 through a PGP were first identified. To collect physician NPIs, we used non-CMS sources, specifically manual website searches of each PGP, the SK&A office-based physician dataset, and the NPI registry. We then assigned NPIs to PGPs based on organization name and address to create the “Other Data Source List” (“ODSL”). CMS subsequently made lists available that included physicians participating in BPCI for 2015 and 2016 (“CMS List”); comparing allowed evaluation of the accuracy of ODSL. We restricted both CMS List and ODSL to PGPs participating in the largest single episode (major joint replacement of the lower extremity) and then compared physician characteristics by linking information from the SK&A database. We performed chi-squared tests to determine whether ODSL-identified physicians were meaningfully different from CMS List-identified physicians.
Results
ODSL included 8,757 physicians, while the CMS List included 11,758 physicians. Sixty-two percent of ODSL physicians were found in the CMS List of participating BPCI physicians, and ODSL contained 46% of BPCI physicians identified in the CMS List (Table 1). Chi-squared tests performed by specialty, geography, and PGP size rejected equivalence of ODSL and CMS list (P<0.001) (Table 2). ODSL was statistically different from the CMS List and had significant limitations in identifying participating BPCI physicians.
Table 1
| CMS List | Not in CMS List | Positive predictive value | |
|---|---|---|---|
| ODSL | 5,456 | 3,301 | 62% |
| Not in ODSL | 6,302 | – | – |
| True positive | 46% | – | – |
PGP, physician group practice; NPI, National Provider Identifiers; CMS, Centers for Medicare and Medicaid Services; ODSL, Other Data Source List.
Table 2
| Physician characteristics | % of ODSL missing from CMS List | % of CMS list missing from ODSL |
|---|---|---|
| Specialty (P<0.001) (top 3 of 83 specialties shown) | ||
| 1. Internist | 43 | 62 |
| 2. Orthopedic surgeon | 20 | 22 |
| 3. Family practitioner | 48 | 68 |
| Avg. of all specialties | 71 [0–100] | 47 [0–100] |
| Geography (P<0.001) | ||
| Northeast (n=8) | 53 | 50 |
| Midwest (n=12) | 42 | 60 |
| South (n=17) | 32 | 49 |
| West (n=13) | 50 | 57 |
| Avg. by region | 44 [32–53] | 54 [49–60] |
| Avg. by state | 43 [3–100] | 61 [21–100] |
| PGP size (P<0.001) | ||
| Large (n=62) (50+ physicians) | 22 [0–100] | 49 [10–88] |
| Medium (n=149) (10–49 physicians) | 53 [0–100] | 27 [0–92] |
| Small (n=77) (<10 physicians) | 74 [0–100] | 14 [0–93] |
CMS, Centers for Medicare and Medicaid Services; ODSL, Other Data Source List; PGP, physician group practice.
Specialty
On average, 71% of ODSL-identified physicians were missing from the CMS List, while 47% of CMS List-identified physicians were missing from ODSL. Of note, 43% of ODSL-identified internists, 20% of orthopedic surgeons, and 25% of physical medicine/rehab specialists were missing from the CMS List.
Geography
On average in each state and region (Northeast, Midwest, South, West), 43% (3–100%) and 44% (32–53%), respectively, of ODSL-identified physicians were missing from the CMS List, while 61% (20–100%) and 54% (49–60%), respectively, of CMS List-identified physicians were missing from ODSL.
PGP size
On average in large (50+ physicians), medium (10–49 physicians), and small PGPs (<10 physicians), 22%, 53%, and 74% (each 0–100%), respectively, of ODSL-identified physicians were missing from the CMS List, while 49% (10–88%), 27% (0–92%), and 14% (0–93%), respectively, of CMS List-identified physicians were missing from ODSL.
Discussion
In this study, we examined physician group participation in Medicare’s BPCI program and found that publicly available data could not be used to accurately identify a large proportion of participants. This suggests that the health policy research community is heavily reliant on the release of such data by regulatory agencies such as CMS to provide policy relevant analysis of program impact. Researchers frequently use primary data collection such as web scraping and other manual means to collect this information, as we did in this study, to fill in gaps in availability of data. While this is generally well-intentioned and noted as a limitation of research, this is the first study to our knowledge to compare a detailed effort to collate participation lists using publicly available data to participation lists made available by CMS. Unfortunately, our analysis did not corroborate that manual efforts are accurate.
This limitation of manual collection may not be limited to the BPCI program. There are several programs, unlike BPCI, for which the participation lists have not been made available to the research community. For example, to date, the Next Generation ACO program and several primary care initiatives do not provide lists with identifiers that can be used reliably by researchers (4-6).
Conclusions
Policy evaluations that rely on identifying physicians using non-CMS sources may have a large degree of inaccuracy. If these challenges extend to other CMS APMs (e.g., Next Generation Accountable Care Organizations), policy evaluations of such programs using non-CMS sources may also have large degrees of inaccuracy. Before CMS continues expanding its bundled payment APM in PGPs, robust research should be conducted to evaluate the effects of BPCI. However, if CMS does not publicly release and update lists of physicians participating in PGPs in a timely fashion, health policy researchers cannot accurately study the impact of BPCI on PGP physicians. Without such research, CMS will be unable to make an evidence-based decision to continue expanding its bundled payment programs. We expect that the effects of BPCI will be different on PGPs than on hospitals, similar to what has been observed in other APMs (7-9).
Acknowledgments
The Perelman School of Medicine at the University of Pennsylvania Health Services Research Summer Scholars Program sponsored by the Department of Medicine in collaboration with the Leonard Davis Institute of Health Economics and the Master of Science in Health Policy Research Program.
Funding: None.
Footnote
Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jhmhp.2019.06.03). Dr. Navathe reports receiving grants from Hawaii Medical Services Association, Anthem Public Policy Institute, Cigna, and Oscar Health; personal fees from Navvis and Company, Navigant Inc., Lynx Medical, Indegene Inc., Sutherland Global Services, and Agathos, Inc.; personal fees and equity from NavaHealth; personal fees from the National University Health System of Singapore; speaking fees from the Cleveland Clinic; serving as a board member of Integrated Services Inc., a subsidiary of Hawaii Medical Services Association, without compensation, and an honorarium from Elsevier Press, none of which are related to this manuscript. The other authors have no conflicts of interest to declare.
Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.
References
- Verma S. “Medicare and Medicaid Need Innovation.” Wall Street Journal [published 19 September 2017, accessed 2 January 2018]. Available online: https://www.wsj.com/articles/medicare-and-medicaid-need-innovation-1505862017
- The Centers for Medicare & Medicaid Services. CMS Announces Participants in New Value-Based Bundled Payment Model. [Published 9 October 2018, accessed 22 October 2018] Available online: https://www.cms.gov/newsroom/press-releases/cms-announces-participants-new-value-based-bundled-payment-model
- The Lewin Group. CMS Bundled Payments for Care Improvement Initiative Models 2-4: Year 3 Evaluation & Monitoring Annual Report. [Published October 2017, accessed 22 October 2018] Available online: https://downloads.cms.gov/files/cmmi/bpci-models2-4yr3evalrpt.pdf
- The Centers for Medicare & Medicaid Services. Comprehensive Primary Care Initiative. [Updated 14 May 2019, accessed 07 June 2019] Available online: https://innovation.cms.gov/initiatives/comprehensive-primary-care-initiative/
- The Centers for Medicare & Medicaid Services. Comprehensive Primary Care Plus. [Updated 14 May 2019, accessed 07 June 2019] Available online: https://innovation.cms.gov/initiatives/comprehensive-primary-care-plus/
- The Centers for Medicare & Medicaid Services. Next Generation ACO Model. [Updated 21 May 2019, accessed 07 June 2019] Available online: https://innovation.cms.gov/initiatives/next-generation-aco-model/
- McWilliams JM, Hatfield LA, Landon BE, et al. Medicare Spending after 3 Years of the Medicare Shared Savings Program. N Engl J Med 2018;379:1139-49. [Crossref] [PubMed]
- Muhlestein D, Saunders RS, McClellan MB. Medicare Accountable Care Organization Results For 2015: The Journey To Better Quality And Lower Costs Continues [published 09 September 2016, accessed 22 October 2018. Available online: https://www.healthaffairs.org/do/10.1377/hblog20160909.056418/full/
- Saunders RS, Muhlestein D, & McClellan MB. Medicare Accountable Care Organization Results For 2016: Seeing Improvement, Transformation Takes Time. (published 21 November 2017, accessed 22 October 2018) Available online: https://www.healthaffairs.org/do/10.1377/hblog20171120.211043/full/
Cite this article as: Isidro U, Martinez JR, Navathe AS. CMS program participation and policy evaluation without administrative data: a case study on Bundled Payments for Care Improvement (BPCI) initiative. J Hosp Manag Health Policy 2019;3:16.
