Using a diversity, equity and inclusion Net Promoter Score to measure confidence in program impact

Introduction

Corporations have routinely maintained a business case for diversity and have traditionally evaluated success by relying on workforce metrics. Organizations that have invested heavily in the recruitment and retention of underrepresented minorities have seen better business performance and a greater return on investment than their peers (1). While quantitative indicators of workforce composition are important, others have argued that the evaluation of diversity initiatives must include measures to assess culture change because a diverse workforce is not necessarily an inclusive or equitable one. They maintain that diversity metrics need to be aligned with key organizational values and goals and employers must seek employee input to understand if efforts have resulted in real culture change (2). Academic Medical Centers have struggled with metrics that go beyond an analysis of workforce composition and typical employee engagement surveys to assess progress with diversity initiatives.

At Michigan Medicine leaders at all levels have embraced the value of an inclusive and collaborative work environment as an avenue to retain a diverse workforce and have sought employee feedback regarding their perception of the real impact of diversity, equity and inclusion (DEI) efforts in their departments. Employees are asked to rate their level of confidence that DEI programming is making a positive impact on culture on a scale of 0–10. While there is no general agreement of the best approach, the direct assessment of confidence has historically been ascertained using a Likert scale with three to seven levels. Although there has not been an examination of a 0–10 scale for confidence, previous methodological research has determined that finer classification levels lead to more variance and less rounding error (3). The net promoter methodology achieves a more sensitive measurement of confidence and provides a common language around program impact. The advantages of using this innovative approach are that it is easy to use and to administer and it allows benchmarking across departments. Using the overall score in isolation is the major disadvantage of the Net Promoter Score (NPS). The score itself does not identify reasons for a lack of confidence. Feedback from groups listed as detractors, passives and promoters must be used to inform strategy to improve programs. We present the following article in accordance with the MDAR reporting checklist (available at http://dx.doi.org/10.21037/jhmhp-19-119).

Methods

A collaboration of employees from the Office for Health Equity and Inclusion, Human Resources, Quality Department, and Organizational Learning designed an integrated DEI dashboard in Tableau that combined institutional data with a DEI NPS designed to measure the impact of local area DEI initiatives.

The DEI dashboard contains four domains populated with institutional and census data: community alignment (census data), workforce demographics and opportunities (institutional human resource data), culture (institutional survey data focused on respect, communication, teamwork, and DEI climate), and DEI implementation (DEI NPS). The tool allows the user to filter down from the institutional view to their specific department and permits views by sex, age, ethnicity, and job codes (where applicable and within limits to ensure confidentiality). The ability to combine measures across domains enables the user to identify patterns where DEI efforts have been successful and to suggest areas where targeted interventions to enhance DEI would be beneficial. Thus, the dashboard facilitates connections between the “what” and “why” of DEI.

The NPS, initially designed as a customer relationship metric (4), has been used effectively to gauge customer satisfaction and loyalty, to identify areas needing improvement, and to promote competition for program improvement. The research behind the NPS methodology shows that companies with scores higher than their competition grow faster and are more successful. Not only has the NPS served as a leading indicator of growth in business, it has also served as an effective tool to engage the workforce. Apple Retail pioneered the employee NPS (eNPS), asking employees “how likely is it that you would recommend this company as a place to work?” (5). Apple quickly realized that employee feedback strengthens larger initiatives and began using the eNPS to genuinely connect with employees by providing them with a voice to increase engagement. The NPS has also been adapted for a medical setting as a discriminator of health care performance (6). Michigan Medicine currently uses NPS in the majority of patient care settings to evaluate patients’ willingness to recommend services or providers to friends or family (7). Thus, having an NPS measure in DEI fosters alignment of metrics across several important domains of the institution.

The simplicity of the NPS allows for easy utilization to measure progress with any initiative. In early 2016, Michigan Medicine adopted the NPS strategy to provide meaningful feedback from customers on the operation of its newly created Human Resources Solutions Center (HRSC). The Center supports 28,000 customers in the areas of compensation, employment, policy interpretation, and employee leaves. A user-friendly and inexpensive feedback mechanism was required to provide meaningful information that could be used to inform future HRSC strategy. The HRSC adopted the NPS philosophy and asked a simple question, “Based on your recent experience with the HRSC, what is the likelihood that you would recommend a colleague use our services?”. Timely customer feedback continually informed strategy relating to incremental improvements in the HRSC throughout the year.

Aligned chronologically with the HRSC transformation was a University of Michigan global initiative related to creating a diverse, equitable, and inclusive environment. To measure progress with mission critical initiatives, the DEI Measurement and Metrics team gathered feedback on DEI plans using an adaptation of the customer satisfaction NPS. NPS was leveraged in a unique way to assess respondents’ views on how institutional culture is being shaped by DEI initiatives. To test the ease of implementation and of use, the NPS methodology was piloted in 2018 with over 100 DEI implementation leads throughout Michigan Medicine, asking the question: “On a scale of zero to ten, based on your experience implementing your unit’s DEI plan, how confident are you that your plan is making a positive impact on culture in your area?”. This was followed up with the question, “Why did you choose that number?”. Three categories of responders were identified and classified: ‘promoters’, those confident that DEI initiatives are resulting in culture change; ‘passives’, those who recognized that efforts were underway but are not convinced of the impact; and ‘detractors’, those who failed to see any impact of DEI initiatives in their areas. The overall DEI NPS score is calculated from the percentage of promoters—percentage of detractors. The NPS can range from –100 to +100. Over 60% of the DEI implementation leads participated in the pilot and initial NPS for units ranged from a low of –60 to –15, and the aggregate score was –45. Feedback from the follow-up question was content-coded and used to identify barriers to plan implementation.

After a successful pilot of the DEI NPS, the wording of the question was changed slightly to include all faculty and staff so that all voices would be represented. The revised DEI NPS question, “How confident are you that your work unit/department’s diversity, equity and inclusion efforts are making a positive impact on culture in your work unit/department?” was added to the 2019 annual employee engagement and 2018 faculty satisfaction surveys. (Both surveys received response rates over 70%). The placement of the DEI NPS within existing institutional surveys provided the opportunity to determine the relationship between the DEI NPS and other key measures of engagement and culture.

The culture measures included three items from the employee engagement survey: (I) teamwork: I receive the necessary support from employees in my unit/department to help me succeed in my work; (II) respect: my ideas and suggestions are valued by my organization, and (III) diversity: my organization understands and respects differences among employees (gender, race, age, religion, etc.).

Employee engagement was measured by responses to four questions (8): (I) this organization inspires me to perform my best; (II) I am willing to put in a great deal of effort in order to help this organization succeed; (III) I would recommend this organization to my friends as a great place to work; and (IV) I am likely to be working for this organization 3 years from now. Questions are rated on a 6-point scale [1] strongly disagree to [6] strongly agree. The average score across the four engagement questions for each participant constitutes the Engagement Index, with a range from 1–6. To be considered “engaged”, participants must answer “strongly agree” to at least two of the four questions and no less than “agree” to any of the four questions.

Statistical analysis

To evaluate differences among the three NPS groups, the Kruskal-Wallis test was used for continuous variables since the normality assumption was not met. Wilcoxon rank sum test was used for pairwise analysis. Chi-square test or Fisher’s exact test was used for categorical variables, and Spearman’s rank correlation coefficient was conducted to assess the relationship between DEI NPS groups and measures of culture, as well as the engagement index. A 5% level of significance was used to evaluate statistical significance in all analyses. Statistical analyses were performed using SAS software, Version 9.4 (SAS Institute Inc., Cary, NC, USA).

Results

Over 18,000 staff members completed the employee engagement survey in 2019. Using the results of the DEI NPS, 28% of employees were classified as promoters, 39.4% as passives, and 32.6% as detractors, making the overall DEI NPS for staff –4.6.

Table 1 illustrates the staff demographics and selected measures in the employee engagement survey by DEI NPS group. The distribution of gender, age group, ethnicity, tenure, and engagement were not homogeneous across DEI NPS groups. Male staff members were classified as promoters more than their female counterparts, (29.7% vs. 27.6%), and significantly more under-representative minorities (URMs) were classified as detractors, compared with all other ethnicities. Additionally, staff with less than 1 year of tenure were more likely to be promoters (33.5%) than any other tenure category.

There was also a clear relationship between DEI Net promoter groups and the measures designed to assess positive culture. Promoters scored significantly higher than passives or detractors on items measuring teamwork, respect, and diversity. All culture items were significantly different among the groups, as well as the pairwise comparisons, with the P values <0.0001 in every pair.

Overall employee engagement varied significantly between DEI Net promoter groups, with 47% of engaged employees classified as promoters in comparison to just 16% of unengaged employees (P<0.0001). Additionally, Spearman’s rank correlation coefficient indicated a statistically significant relationship between NPS groups and the employee engagement index (P<0.0001).

The strong correlation between the NPS and employee engagement lead us to examine this relationship more closely. A multiple logistic regression analysis using backward model selection was conducted to determine if, after controlling for demographic variables and culture measures, the DEI NPS predicts employee engagement. Results are presented in Table 2. Overall, a significant association was found between engaged employees and sex, age, URM status, tenure, culture measures, and net promoter categories. Female respondents were 1.3 times more likely to be considered engaged employees than their male counterparts. Compared to younger employees (<25 years of age), the relationship between age and engagement appears to get stronger with increasing age. Using Caucasian ethnicity as the reference category, the odds for engagement for URM are significant at 1.2. However, the results for Asian and two or more races, compared to Caucasian, are not significant predictors of engagement. Newly hired employees are the most engaged employees in this sample. Compared to those who have been with the organization 15 years or more, they are 1.4 times more likely to be engaged. The relationship between the culture measures and engagement are consistent with literature suggesting that employees experiencing positive culture are more likely to be engaged (9,10). After controlling for these culture measures, tenure, and demographics, promoters were over two times more likely to be classified as engaged employees than detractors.

Discussion

Here we have described a novel, inexpensive, highly visual, and simple to understand metric for measuring perceived impact of DEI efforts in an Academic Medical Center. The DEI NPS provides a key measure of employee confidence in DEI initiatives. A single score cannot provide all the information needed to guide strategic improvements. However, when complemented with feedback and other measures of employee engagement, it provides a concise measure of the organization’s DEI performance through the eyes of its employees. Through feedback regarding the DEI NPS, significant insight was gained and guided strategic improvements in DEI efforts. At the organizational level, feedback demonstrated that several leaders need assistance rolling out new DEI approaches and, as a result, short 3-minute videos with conversation questions were released as toolkits (11). In response to the need to diversify the workforce, the use of standardized behavioral interview questions aligned with the University of Michigan’s Expectation Model are being employed throughout the organization. Additionally, for current staff, LinkedIn Learning (12) content focused on building trust and encouraging open communication has been leveraged and offered to all benefits-eligible faculty and staff at no cost. To understand why people voluntarily leave the organization, exit interview focus groups have been initiated and are expanding. All of these efforts have leveraged the knowledge gained through NPS feedback. Local units have not only been able to tailor their DEI programs to the unique needs of their staff members, but they have also been able to gauge impact quickly and efficiently.

Conclusions

As organizational diversity initiatives continue to expand, perception of impact will be critical to measuring success. The DEI NPS is one simple measure of perceived impact that aligns well with measures of positive culture and predicts employee engagement. It engages all employees to assess the progress of culture change and is one of many measures used to determine how well Michigan Medicine is living the mission of DEI. The DEI NPS will continue to be included in established institutional surveys so that we have a timely assessment of how all employees feel about DEI programming and it will continue to inform strategy. An engaged workforce that has confidence in the impact of the organization’s commitment to diversity can only strengthen the business case for diversity and contribute to an inclusive environment.

Acknowledgments

Funding: None.

Footnote

Reporting Checklist: The authors have completed the MDAR reporting checklist. Available at http://dx.doi.org/10.21037/jhmhp-19-119

Data Sharing Statement: Available at http://dx.doi.org/10.21037/jhmhp-19-119

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jhmhp-19-119). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Herring C. Does Diversity Pay?: Race, Gender, and the Business Case for Diversity. Am Sociol Rev 2009;74:208-24. [Crossref]
2. Herdman AO, McMillan-Capehart A. Establishing a diversity program is not enough: exploring the determinants of diversity climate. J Bus Psychol 2010;25:39-53. [Crossref]
3. Green PE, Rao VR. Rating scales and information recovery—how many scales and response categories to use. J Mark 1970;34:33-9.
4. Reichheld FF. The one number you need to grow. Harv Bus Rev 2003;81:46-54, 124. [PubMed]
5. Reichheld FF. The Ultimate Question 2.0: How Net Promoter Companies Thrive in a Customer-Driven World. Brighton: Harvard Business Review Press, 2011.
6. Quality Department at Michigan Medicine. Available online: https://quality.i.medicine.umich.edu/divisions/quality-analytics/patient-satisfaction
7. Press Ganey Associates. About Press Ganey. Available online: https://www.pressganey.com/about
8. Hamilton DF, Lane JV, Gaston P, et al. Assessing treatment outcomes using a single question: the Net Promoter Score. Bone Joint J 2014;96-B:622-8. [Crossref] [PubMed]
9. Jagannathan A. Determinants of employee engagement and their impact of employee performance. International Journal of Productivity and Performance Management 2014;63:308-23. [Crossref]
10. Geue PE. Positive practices in the workplace: impact on team climate, work engagement, and task performance. J Appl Behav Sci 2018;54:272-301. [Crossref]
11. Robbins SL. Associates, Inclusion Insights: Burst Learning Video series. Available online: https://www.slrobbins.com/inclusion-insights
12. LinkedIn Learning. Available online: https://www.linkedin.com/learning