Evaluating the importance of digital tools or approaches to hospital performance using the analytic hierarchy process (AHP)-Delphi approach

Introduction

Research background

Hospital performance indicators

Organizational performance is the achievement of an organization including financial and nonfinancial indicators (1,2). Digitalization, in this study, is defined as the use of digital technologies for processing and delivery of healthcare services, as adapted from the European Commission (3). Previous studies in digital hospitals mostly focused on the applications of digital tools or approaches in hospital operations. Thus, hospital performance is measured mainly by operational indicators such as clinical effectiveness (4,5), business processes (4,6,7), integrated care based on communications and collaboration between healthcare professionals and patients (8,9), efficiency and/or effectiveness (10,11). Paul et al. (7) proposed patient satisfaction indicators along with business process indicators. Table 1 presents performance indicators of general and digitalized hospitals. For nonfinancial indicators, the performance of digitalized hospitals is mostly measured by operational indicators (business processes, integrated healthcare, healthcare quality, efficiency, and long-term development) and marketing indicators (patient satisfaction). In the meantime, financial dimensions have not been mentioned in previous studies but they should be included in performance indicators for digitalized hospitals (17,18) since they are the basic factors for hospital development. Thus, those financial performance indicators will be re-assessed and confirmed by hospital managers in this study. In general, there is a dearth of research that ranks and evaluates hospital performance indicators using expert opinion, even though hospital management must take these indicators into account when creating strategic plans to enhance hospital performance.

Digital tools for hospitals

The selection of digital tools or approaches for hospitals depends upon the objectives, strategy, digitalization budget, and staff abilities (3,19). Given the hospital operations, the applications of digital tools or approaches can be classified into 2 groups: (I) patient-centricity support, and (II) hospital management support. Those tools or approaches can support hospital management, leverage the interactions between patients and doctors or among doctors, and facilitate clinical activities in diagnosis, and healthcare treatment for quality improvement (16,19,20). The classification of digital tools or approaches is presented in Table 2. While previous studies have made significant contributions to the digitalization of hospitals, they have primarily concentrated on the use of digital tools or approaches to improve hospital operations. Research on the evaluation and ranking of digital tools or approaches for use in hospitals to improve hospital performance indicators is still in shortage.

The contribution of digital tools to hospital performance

Due to the existence of many digital tools or approaches, hospital managers need to consider their contribution to hospital performance (21). Among the tools or approaches, telemedicine uses electronic information and communications technologies to provide and support healthcare from a distance (22,23). Thus, telemedicine can contribute to the workload reduction of hospitals and improve healthcare quality and patient satisfaction. The online interactions between patients-doctors or among doctors help to leverage integrated healthcare and select the most suitable treatment approach for the patients (24). One of the main objectives of hospital digitalization is to increase the connection capability among healthcare institutions for integrated healthcare and supply chain effectiveness (25). The digital tool or approach helps transfer patient data, image diagnostics, and clinical exams from one hospital to other healthcare institutions for integrated healthcare. This connection can also increase interactions between hospitals and suppliers for efficient hospital supply chain management (21). This results in clinical effectiveness, healthcare quality improvement, and patient satisfaction. The other digital tools or approaches are also claimed to greatly contribute to hospital performance. Online real-time medicine delivery gives instructions, and consultancy to the patients in using the medicine. Hence, this facilitates the medicine provision to the patients on time, resulting in healthcare quality and patient safety (26). Online real-time patient scheduling is very important to reduce hospital overload and patient waiting time. With the support of artificial intelligence (AI), the average time that a patient spends at registration and medical examinations can be estimated and forecasted (10,27). This helps to balance the hospital capacity and the workload, hence, increasing the healthcare quality and patients’ satisfaction. Electronic medical records (EMRs) facilitate the data availability to hospital managers and doctors for their decision-making, hence, improving working capacity, and reducing the workload (28). To store the clinical diagnosis data in EMRs, hospitals should implement Laboratory Information System (LIS), Radiology Information System (RIS), then, Picture Achieving and Communication System (PACS) to facilitate data storage and sharing, which makes the data available for doctors to propose medical treatment (29). It is obviously stated that digital tools or approaches can help to solve critical problems in healthcare management and improve hospital performance, for example, providing medicine to the patients on time, improving working capacity and reducing the workload, and facilitating integrated healthcare (21,24,26,28). However, up to date, there is still a lack of research in classifying and prioritizing the significant digital tools or approaches for hospital performance improvement in both breadth and depth. Thus, hospital managers will subjectively select tools or approaches to implement resulting in ineffective hospital digitalization and unsolvable problems such as overloading, long waiting times, low healthcare quality, low patient satisfaction, and low hospital performance.

Research rationale and objectives

Hospital digitalization has become an emerging topic in academic research and application for healthcare management (30-32). Digital tools or approaches applied in hospitals help manage patient registration, diagnosis, treatment, and healthcare supply chain (16,20,33,34). As a result, digital hospitals can benefit from reducing patient waiting time and cost and improving hospital performance [healthcare quality, patient satisfaction, and hospitals’ efficiency and long-term development (11,35)]. Digitalization can be a strategy to improve hospital performance and enahnce patient service (19). Thus, hospital managers should define dimensions of hospital performance that can be improved by applying digital tools or approaches (36). Unfortunately, state-of-the-art studies have not focused on exploring the role of digital tools or approaches in improving hospital performance indicators (37), especially in developing countries. In practice, hospital digitalization projects are proceeded with high implementation cost, but unexpected performance (38) because of lacking suitable tools or approaches for digital transformation. This problem can be more serious for hospitals in developing countries like Vietnam. This is because, in low and middle-income countries, hospitals have limited financial resources, inadequate technology infrastructure, insufficient planning, shortage of skilled personnel, unavailability of data, and managers’ low awareness of digitalization (39,40). Thus, to fill the research gap in hospital digitalization and help hospital managers overcome the above-stated problem, this study is conducted to achieve the following objectives: (I) recognizing the important indicators of digital hospital performance; (II) using these indicators as strategic criteria for the evaluation and prioritization of digital tools or approaches applied for hospitals; and (III) proposing the managerial implications to enhance hospital performance through applying digital tools or approaches.

Methods

Following the instructions of Jünger et al. (41) for reporting the Delphi study, our research method is presented as the following:

Purpose and rationale

It is worthwhile to pay attention to experts from hospitals and universities to develop preliminary ideas for the subject, as there is currently insufficient research examining the importance of digital tools or approaches in enhancing hospital performance. Experts can address complicated problems on an individual basis through the Delphi process with structured group communication (42). A Delphi study requires a minimum of 20 participants to prevent individual biases from impacting the outcome of group communication (42).

Expert panel

Academic researchers from universities, hospital managers, and healthcare professionals with expertise in hospital management were invited to participate in the Delphi procedure for this study. To represent the healthcare professionals, this study invited doctors since doctors, as their delegated functions, have to use digital tools or approaches to work closely with patients to consult or treat patients’ diseases, hence, having a deep understanding of patient expectations. Hence, doctors can apply these tools or approaches to serve and satisfy their patients. Nurses collaborate with doctors to take care of the patients in medicines offering, usage, and monitor patients’ vital signs, injections, and infusion … following doctors’ instructions. Thus, doctors, who normally know all actions, and issues in nurses’ activities, can represent healthcare professionals (43). Patients are beneficiaries of hospital digitalization strategy and their demands are well understood by healthcare professionals. As such, patients are not invited to this survey.

The following criteria were used to choose these participants: (I) possessing a minimum of 5 years of working or hospital digitalization experience; (II) possessing extensive understanding regarding healthcare management and digitization; (III) being open to participating in this study. Table 3 displays the specifics of these individuals.

Description of integrated analytic hierarchy process (AHP)-Delphi method

This study employed the Delphi technique to determine the primary hospital performance indicators, which serve as strategic standards for assessing the digital tools or approaches for hospital administration. Thus, to rate the value of digital tools or approaches, this study requires a multi-criteria decision-making process in addition to the Delphi method. The AHP employs a multi-criteria decision-making technique that streamlines item evaluation and ranking via ratio-scale pairwise comparisons (44). To accomplish the research objectives in this study, the Delphi method and the AHP approach are combined.

The following is a presentation of the AHP pair-wise comparison computing process: let n be the number of criteria or factors. The matrix of pairwise comparison is displayed as follows:

A=[aij]=(a11...a1n.........an1...ann)(i,j=1,2,...,n)[1]

where a_ij presents the ratio scale of the score provided by experts for factors i and j; a_ij >0; a_ij =1/a_jiwhen i ≠ j; a_ij =1 when i = j. The mean values of a_ijare computed by:

W¯i=1n∑j=1naij(i=1,2,...,n)[2]

The vector of weight W = (W₁,…, W_n)^T is defined as the normalization of the vector W¯=(W1¯,...,Wn¯)T:

Wi=W¯i∑j=1nW¯j(j=1,2,...,n)[3]

A consistency index (CI) is used to examine the consistency of the matrix with a large order n. The CI and the maximal eigenvalue of the matrix are determined by:

λmax=1n∑i=1n(AW)iWi[4]

CI=λmax−nn−1[5]

If the CI value is small enough, consistency is assured. Since random factors may contribute to a consistency deviation, it is necessary to determine the consistency ratio (CR):

CR=CIRI[6]

where the random CI is denoted by RI. For various indexes n =(1; 2; 3; 4; 5; 6; 7; 8; 9; 10) accordingly, the values of RI =(0.000; 0.000; 0.580; 0.090; 1.120; 1.240; 1.320; 1.410; 1.450; 1.490) are standardized. The consistency level is acceptable if CR is less than 0.100; however, if CR is greater than 0.100, it is advised to indicate significant inconsistencies.

For the AHP-Delphi process, a conference was organized to attain two goals: (I) to identify and weigh the most important hospital performance indicators, and (II) to use these performance indicators as strategic criteria to determine how much digital tools or approaches contribute to hospital performance.

Procedure of AHP-Delphi

The conference is run using the following rounds to accomplish these goals:

• Round 1: describe digitalization and hospital performance in more detail: to guarantee accurate comprehension of the terms, the hosts provided a quick overview of digitalization and hospital performance to all participants. This explanation was often provided in the synopsis of this article’s prior part, which was the literature review.
• Round 2: identify key hospital performance indicators: the evaluation sheets with key hospital performance indicators that were produced based on the literature review were given to the expert panel. The participants were asked to rank the importance of these indicators concerning hospitals’ strategic development. The 5-point scale (from “1: not important” to “5: extremely important”) was used for the evaluations. Expert feedback was anonymous and based on individuals, which allowed for the utilization of a variety of feedback content and prevented group discussion biases (45). The interquartile deviation (IQD) was applied following Rayens and Hahn’s (46) instructions to reach the evaluation consensus. An IQD of 1.000 or below is regarded as a consensus indicator for the 5-point scaled assessments (47). Table 4 displays the outcome of this round, which indicates that the evaluations’ IQDs for 7 indicators are less than 1.000. This suggests that the consensus is reached in the assessments made by each panel expert.
• Round 3: evaluate the weights of key hospital performance indicators: in this round, the experts were requested to compare the importance levels of the 7 hospital performance indicators in pairs as a preliminary step in the AHP process. The following pairwise comparisons were carried out using Saaty’s (55) 9-point scale (from “1: equal importance: both elements have equal contribution to the objective” to “9: extreme importance: one element is favored in comparison with the other based on strongly proved evidence and facts”). For AHP analysis, the program Expert Choice processes data that experts provide. The CR (0.022) is less than 0.100, indicating consistency among participants’ ratings. Table 5 presents the weights and rankings of the hospital performance indicators.
• Round 4: evaluate the weights of digital tools or approaches using hospital performance indicators as strategic criteria: the experts received the prepared evaluation sheets, which included 10 digital tools or approaches, based on the literature review. While the literature review lists a plethora of potential digital tools or approaches for hospital management, only 10 digital tools or approaches, which are divided into two categories: patient-centricity support and hospital management support, were chosen for evaluation in this study because prior research has deemed them to be effective hospital practices (16,19). Seven strategic criteria for comparing digital tools or approaches were derived from the 7 hospital performance indicators that were selected. The Saaty’s (55) 9-point scale was also used for the pairwise comparisons. The AHP structure used for this study is presented in Figure 1.
  • Expert Choice software was also used to process expert data for AHP analysis. Through pairwise comparison, the weights of all digital tools contributing to each performance indicator were calculated. As seen by Table 6, all performance indicators have CRs of less than 0.100, indicating that expert assessments are consistent with one another.
  • The expert panel was asked to approve the AHP findings. A qualitative method known as “post-group consensus” ensured the unanimity for this round. According to von der Gracht (47), this method assesses how much each panelist has personally agreed with the panel’s overall decision following this round. Every panelist, who was asked to explain the outcome on an individual basis, agreed with the outcome.
  • Round 5: result discussion: the purpose of this last round was to listen to the experts’ opinions on the findings, particularly their comments regarding the contributions of effective digital tools or approaches to performance indicators. Their ideas were used as the base for the managerial implications for managers in digital tool or approach applications.

Figure 1 The AHP structure used for this study. PACS, Picture Achieving and Communication System; LIS, Laboratory Information System; RIS, Radiology Information System; HL7, Health Level Seven; AHP, analytic hierarchy process.

Consensus attainment

The criteria for statistical consensus and expert agreement are clearly presented in the above rounds of the AHP-Delphi procedure.

Ethical statement

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Review Board of School of Medicine-Vietnam National University Ho Chi Minh City (No. 8/QD-IRB-VN01.017) and informed consent was obtained from all individual participants.

Results

Key hospital performance indicators

As mentioned above, the weights of 7 hospital performance indicators are presented in Table 5. The indicator of healthcare quality has the highest score (0.233) which is followed by the other indicators: patients’ satisfaction (0.231), integrated healthcare (0.208), business processes (0.117), the long-term development of the hospital (0.079), efficiency in hospital’s resource exploitation (0.073), and financial performance (0.059). The IQD was applied following Rayens and Hahn’s (46) instructions to reach the evaluation consensus. Table 4 displays that the evaluations’ IQDs for 7 indicators are less than 1.000 ensuring the evaluation consensus of all the experts. The experts have the highest consensus that healthcare quality is the most important performance that every hospital must take into account for its long-term development. They believe that healthcare quality is the most important indicator that they have to ensure because the main function of hospitals is to provide healthcare to recover the patient’s health and save the patient’s life as per the Ministry of Health’s requirement. Specifically, the expert A, a hospital manager, stated that “Hospital management must give careful consideration to the matter of enhancing the quality of medical examination and treatment by putting into practice a comprehensive and rigorous set of quality standards and criteria since this is recognized as a critical responsibility for the development of hospitals”. The expert B, also a hospital manager, agreed with this statement by expressing that “Hospitals need to commit to providing safe and high-quality medical services to the patients”. In Vietnam, most of the hospitals are overloaded. Hence, patients are underserved with long waiting times and short-time interactions with doctors. As such, patient satisfaction is low and needs to be improved; thus, it becomes the second important indicator. The experts claimed that integrated healthcare must be considered among hospitals as a requirement by the Ministry of Health to improve the capability of all hospitals. Also, because of the need for healthcare quality improvement, the business processes are of the fourth concern. Hospitals in Vietnam have been transformed from paper-based hospitals to digital hospitals. According to the experts, to serve continuously patients with improved healthcare quality and patient satisfaction, long-term development should be the fifth concern of hospital managers. In the meantime, hospital financial performance is the last concern of hospital managers because most public hospitals in Vietnam follow the Ministry of Health’s requirement to serve patients and the community with the best healthcare quality without considering treatment cost and financial performance.

Digital tools or approaches contribute to hospital performance

Table 6 presents the weights of all evaluated digital tools or approaches which are grouped into two categories: (I) patient centricity support, and (II) hospital management support. Based on the results, the experts suggest that the tools or approaches, that receive the weight scores of 0.100 or more, are considered as the ones having important contributions to hospital performance.

An important objective of this study is to evaluate the weights of digital tools or approaches contributing to 7 identified hospital performance indicators. Thus, an overall evaluation of these digital tools or approaches is significant to recommend the managerial implications for managers in applying digital tools or approarches to enhance hospital performances. Overall, telemedicine (0.164), online real-time interactions (0.152), 4.0 medical infrastructure (0.123), and tools or approaches for connecting with other outside institutions’ IT systems (0.102) are the most important digital tools or approaches that highly contribute to all hospital performance indicators in terms of patient centricity support. The experts agree that these tools or approaches are very necessary to facilitate patient centricity. Telemedicine can help to monitor real-time patients’ health and give instructions to them. Its implementation accelerates hospitals’ business process establishment and leverages integrated healthcare. It also saves patients’ time, protects their health efficiently, and leads to patient satisfaction. Meanwhile, online real-time interactions help give instructions to patients during the treatment process and listen to their health statements to adjust prescriptions to make the treatment process more effective. The weights of these tools or approaches also reveal the expectation of healthcare professionals and patients in using these digital tools or approaches. Since Vietnamese hospital digitalization is in the infancy stage, healthcare professionals have difficulties using these tools or approaches. Healthcare professionals normally have low IT skills and hospitals have low levels of systems integration. It takes a long time for administrative work when the treatment workload is high. Healthcare professionals expect to reduce their workload to have more time to improve their abilities in using digital tools or approaches. Telemedicine is the best way to reduce the workload. In telemedicine, simple tools for online real-time interactions like apps, text messages, and video-based calls are used widely to interact, monitor, and care for patients. Because healthcare professionals prefer telemedicine and online real-time interactions, these tools or approaches have the highest weight. The results align with the findings of previous studies (56,57). Due to a shortage of financial investment, 4.0 medical infrastructure is limited in Vietnamese hospitals; hence, their healthcare technologies and quality have not yet met patient’s demands. Healthcare professionals expect to have the 4.0 medical infrastructure to improve their activities and meet patient’s demands. The results are in confirmation with previous studies on healthcare digital competencies (58,59). Thus, the 4.0 medical infrastructure has the third weight.

In addition, the connection with other healthcare organizations is required by the Ministry of Health to support integrated healthcare. However, in Vietnam, these connections have not been well conducted due to the differences in IT systems and data formats among hospitals.

For hospital management support, PACS, LIS, RIS (0.106), and EMRs (0.095) are considered the most important tools or approaches. The experts claimed that PACS, LIS, RIS [Health Level Seven (HL7)], and EMR are the basis for hospital digitalization to store and transfer patient data inside or outside hospitals, which are required by the Ministry of Health. Those tools or approaches can enhance integrated healthcare, and healthcare quality, as well as reduce the use of printing materials in diagnosis and examination. These tools or approaches when combined with Hospital Information System (HIS) and/or AI tools or approaches (like data mining, or machine learning) can help hospitals optimize their capacity and efficiency in infrastructure usage.

Discussion

The performance indicators identified in this study cover the full scope of hospital performance: operational indicators (healthcare quality, integrated healthcare, business processes, efficiency, long-term development), marketing indicator (patient satisfaction), and finance indicator (financial performance). This result aligns with the organizational performance framework suggested by Chow and Van der Stede (1) and Chenhall and Langfield-Smith (2) as well as contributes to the performance measures of digitalized hospitals which adds value to previous research (4,7,10,16,27).

This study also identifies 10 digital tools or approaches that need implementation for hospital digitalization. Digital tools or approaches (telemedicine, online real-time interactions between patients and doctors, 4.0 medical infrastructure, and tools or approaches for connecting with other outside institutions’ IT) transfer the patient’s role from inactivity to activity. Thus, they facilitate interactions between patients and doctors for treatment with a short time of waiting, and low cost. Hence, they support patient centricity (60) and can be classified as digital tools or approaches supporting patient centricity (61). The other tools or approaches (PACS, LIS, RIS, HL7), and EMRs are confirmed to support the hospital management. Given their contribution to digitalized hospital performance, the results present a high ranking for these digital tools or approaches. This encourages hospital managers to select suitable tools or approaches for higher hospital performance. Since there are very limited studies exploring the importance of digital tools or approaches in enhancing hospital performance (33), the results contribute to filling this gap. Thus, they are very helpful for hospital digitalization, particularly, in developing countries like Vietnam with a shortage of financial resources. Based on the results, hospital managers can define the priority to implement the selected digital tools or approaches and build a roadmap to digitalize hospitals. Hospital managers can be more active in establishing and controlling the digitalization process to achieve expected performance. In addition, hospital digitization success should be predicated on investing in IT infrastructure and training healthcare personnel in digital competencies.

Conclusions

Digitalization is of importance to hospital performance indicators. Since many digital tools or approaches can be implemented, hospital managers need to select suitable ones to digitalize hospitals within the budget limitation to achieve expected hospital performance. Previous studies in this aspect are limited; hence, the results of this study contribute to the knowledge body of hospital digitalization. The results show that hospital performance can be measured with 7 performance indicators which can be improved by the application of 10 digital tools or approaches. The results help hospital managers understand the importance of each digital tool or approaches to the hospital’s performance. Hence, they can have a suitable selection of digital tools or approaches to enhance hospital performance. The generalization of the research results is limited because of the small sample size of invited experts, especially, healthcare professionals. Further research can enlarge the sample size with hospitals of all scales, different departments of the hospitals, and different kinds of healthcare professionals to have more generalized results.

Acknowledgments

We would like to thank Ho Chi Minh City University of Technology (HCMUT), VNU-HCM for the support of time and facilities for this study.

Funding: This study was funded by Vietnam National University-Ho Chi Minh City (VNU-HCM) under grant number DS 2022-20-08.

Footnote

Data Sharing Statement: Available at https://jhmhp.amegroups.com/article/view/10.21037/jhmhp-24-60/dss

Peer Review File: Available at https://jhmhp.amegroups.com/article/view/10.21037/jhmhp-24-60/prf

Conflicts of Interest: Both authors have completed the ICMJE uniform disclosure form (available at https://jhmhp.amegroups.com/article/view/10.21037/jhmhp-24-60/coif). Both authors report the support of time and facilities for this study from Ho Chi Minh City University of Technology (HCMUT), Vietnam National University Ho Chi Minh City (VNU-HCM); and that this study was funded by Vietnam National University Ho Chi Minh City (VNU-HCM) under grant number DS 2022-20-08. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Review Board of School of Medicine-Vietnam National University Ho Chi Minh City (No. 8/QD-IRB-VN01.017) and informed consent was obtained from all individual participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Chow CW, Van der Stede WA. The Use and Usefulness of Nonfinancial Performance Measures. Management Accounting Quarterly 2006;7:1-8.
2. Chenhall RH, Langfield-Smith K. Multiple perspectives of performance measures. European Management Journal 2007;25:266-82. [Crossref]
3. European Commission. Assessing the impact of digital transformation of health services. 2019. Available online: https://health.ec.europa.eu/system/files/2019-11/022_digitaltransformation_en_0.pdf
4. Aarts J, Ash J, Berg M. Extending the understanding of computerized physician order entry: implications for professional collaboration, workflow and quality of care. Int J Med Inform 2007;76:S4-13. [Crossref] [PubMed]
5. Agarwal R, Angst CM, DesRoches CM, et al. Technological viewpoints (frames) about electronic prescribing in physician practices. J Am Med Inform Assoc 2010;17:425-31. [Crossref] [PubMed]
6. Ash JS, Sittig DF, Poon EG, et al. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14:415-23. [Crossref] [PubMed]
7. Paul S, Riffat M, Yasir A, et al. Industry 4.0 applications for medical/healthcare services. J Sens Actuator Netw 2021;10:43. [Crossref]
8. Beuscart-Zéphir MC, Pelayo S, Anceaux F, et al. Impact of CPOE on doctor-nurse cooperation for the medication ordering and administration process. Int J Med Inform 2005;74:629-41. [Crossref] [PubMed]
9. Oborn E, Barrett M, Davidson E. Unity in Diversity: Electronic Patient Record Use in Multidisciplinary Practice. Information Systems Research 2011;22:547-64. [Crossref]
10. Kim SH, Song H. How Digital Transformation Can Improve Hospitals' Operational Decisions. Harvard Business Review 2022;
11. Härkönen H, Lakoma S, Verho A, et al. Impact of digital services on healthcare and social welfare: An umbrella review. Int J Nurs Stud 2024;152:104692. [Crossref] [PubMed]
12. Campbell SM, Roland MO, Buetow SA. Defining quality of care. Soc Sci Med 2000;51:1611-25. [Crossref] [PubMed]
13. WHO. Performance Assessment Tool for Quality Improvement in Hospitals (PATH). 2007. Available online: https://www.who.int/europe/publications/i/item/EUR-04-5064212
14. Smith PC, Mossialos E, Papanicolas I, et al. Performance measurement for health system improvement: experiences, challenges and prospects. WHO European Ministerial Conference on Health Systems. 2010.
15. Bergeron BP. Performance management in healthcare. 2nd edition. CRC Press; 2018.
16. Koebe P, Bohnet-Joschko S. The Impact of Digital Transformation on Inpatient Care: Mixed Methods Study. JMIR Public Health Surveill 2023;9:e40622. [Crossref] [PubMed]
17. Thouin MF, Hoffman JJ, Ford EW. The effect of information technology investment on firm-level performance in the health care industry. Health Care Manage Rev 2008;33:60-8. [Crossref] [PubMed]
18. Ward MJ, Froehle CM, Hart KW, et al. Transient and sustained changes in operational performance, patient evaluation, and medication administration during electronic health record implementation in the emergency department. Ann Emerg Med 2014;63:320-8. [Crossref] [PubMed]
19. WHO. Global strategy on digital health 2020-2025. 2021. Available online: https://www.who.int/publications-detail-redirect/9789240020924
20. Angerer A, Stahl J, Krasniqi E, et al. The Management Perspective in Digital Health Literature: Systematic Review. JMIR Mhealth Uhealth 2022;10:e37624. [Crossref] [PubMed]
21. Beaulieu M, Bentahar O. Digitalization of the healthcare supply chain: A roadmap to generate benefits and effectively support healthcare delivery. Technological Forecasting and Social Change 2021;167:120717. [Crossref]
22. Institute of Medicine (US) Committee on Evaluating Clinical Applications of Telemedicine, Field MJ. editors. Telemedicine: A Guide to Assessing Telecommunications in Health Care. Washington, DC, USA: National Academies Press; 1996.
23. Collins L, Johnson M, Tyson C. The fundamentals of telehealth in population health management. J Hosp Manag Health Policy 2023;7:3. [Crossref]
24. Chen S, Guo X, Wu T, et al. Exploring the Online Doctor-Patient Interaction on Patient Satisfaction Based on Text Mining and Empirical Analysis. Information Processing & Management 2020;57:102253. [Crossref]
25. ENISA. European Union Agency for Network and Information Security. 2018. Available online: https://www.enisa.europa.eu/
26. International Pharmaceutical Federation (FIP). Advancements in digital pharmacy post COVID-19: Report from the FIP Technology Advisory Group. The Hague: International Pharmaceutical Federation. 2023.
27. Klumpp M, Hintze M, Immonen M, et al. Artificial Intelligence for Hospital Health Care: Application Cases and Answers to Challenges in European Hospitals. Healthcare (Basel) 2021;9:961. [Crossref] [PubMed]
28. Bercaw RG, Knoth KA, Snedaker ST. The lean electronic health record: A journey toward optimized care. CRC Press; 2018.
29. Balogh EP, Miller BT, Ball JR, et al. Improving Diagnosis in Health Care. Washington, DC, USA: National Academies Press; 2015.
30. Marques ICP, Ferreira JJM. Digital transformation in the area of health: systematic review of 45 years of evolution. Health and Technology 2020;10:575-86. [Crossref]
31. Tanniru M. Engagement leading to empowerment—digital innovation strategies for patient care continuity. J Hosp Manag Health Policy 2019;3:28. [Crossref]
32. Leggat SG, Yap K. How are hospitals using artificial intelligence in strategic decision-making?—a scoping review. J Hosp Manag Health Policy 2020;4:39. [Crossref]
33. Tortorella GL, Fogliatto FS, Espôsto KF, et al. Measuring the effect of Healthcare 4.0 implementation on hospitals’ performance. Production Planning & Control 2022;33:386-401. [Crossref]
34. Khoja A, Magsi S, Shahnawaz S. “mHealth”—an important role in facilitating primary healthcare. J Hosp Manag Health Policy 2018;2:6. [Crossref]
35. Tortorella GL, Fogliatto FS, Kurnia S, et al. Healthcare 4.0 digital applications: An empirical study on measures, bundles and patient-centered performance. Technological Forecasting and Social Change 2022;181:121780. [Crossref]
36. Huebner C, Flessa S. Strategic Management in Healthcare: A Call for Long-Term and Systems-Thinking in an Uncertain System. Int J Environ Res Public Health 2022;19:8617. [Crossref] [PubMed]
37. Brenner M, Weir A, McCann M, et al. Development of the key performance indicators for digital health interventions: A scoping review. Digit Health 2023;9:20552076231152160. [Crossref] [PubMed]
38. Raimo N, De Turi I, Albergo F, et al. The drivers of the digital transformation in the healthcare industry: An empirical analysis in Italian hospitals. Technovation 2023;121:102558. [Crossref]
39. Chanh HQ, Ming DK, Nguyen QH, et al. Applying artificial intelligence and digital health technologies, Viet Nam. Bull World Health Organ 2023;101:487-92. [Crossref] [PubMed]
40. Mwanza J, Telukdarie A, Igusa T. Impact of industry 4.0 on healthcare systems of low- and middle- income countries: a systematic review. Health Technol (Berl) 2023;13:35-52. [Crossref] [PubMed]
41. Jünger S, Payne SA, Brine J, et al. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med 2017;31:684-706. [Crossref] [PubMed]
42. Akkermans HA, Bogerd P, Yücesan E, et al. The impact of ERP on supply chain management: Exploratory findings from a European Delphi study. European Journal of Operational Research 2003;146:284-301. [Crossref]
43. Wosny M, Strasser LM, Hastings J. Experience of Health Care Professionals Using Digital Tools in the Hospital: Qualitative Systematic Review. JMIR Hum Factors 2023;10:e50357. [Crossref] [PubMed]
44. Jayakumar V, Raju R, Mariappan C, et al. An analytic hierarchical approach to decision making for selection of engineering colleges in Tamil Nadu. IUP Journal of Operations Management 2010;9:16-22.
45. McGrath JE. Groups: Interaction and Performance. Englewood Cliffs, NJ, USA: Prentice-Hall; 1984.
46. Rayens MK, Hahn EJ. Building consensus using the policy Delphi method. Policy, Politics, & Nursing Practice 2000;1:308-15. [Crossref]
47. von der Gracht HA. Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting and Social Change 2012;79:1525-36. [Crossref]
48. Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, USA: National Academies Press; 2001.
49. Linder-Pelz SU. Toward a theory of patient satisfaction. Soc Sci Med 1982;16:577-82. [Crossref] [PubMed]
50. Dawda P. Integrated healthcare: the past, present and future. Integr Healthc J 2019;1:e000001. [Crossref] [PubMed]
51. Buttigieg SC, Dey PK, Gauci D. Business process management in health care: current challenges and future prospects. Innovation and Entrepreneurship in Health 2016;3:1-13. [Crossref]
52. McLean GN. Organizational development: Principles, processes and performance. Berrett-Koehler Publishers; 2005.
53. National Academies of Sciences, Engineering, and Medicine; Health and Medicine Division; Board on Health Care Services; et al. Crossing the Global Quality Chasm: Improving Health Care Worldwide. Washington, DC, USA: National Academies Press; 2018.
54. Gitman LJ, Zutter CJ. Pricinples of managerial finance. 14th edition. Pearson; 2015.
55. Saaty TL. The Analytic Hierarchy Process. New York, NY, USA: McGraw-Hill; 1980.
56. Andronic O, Petrescu GED, Artamonov AR, et al. Healthcare Professionals’ Specialists’ Perception of Telemedicine in Romania—A Quantitative Study of Beliefs, Practices, and Expectations. Healthcare 2023;11:1552. [Crossref] [PubMed]
57. Lundereng ED, Nes AAG, Holmen H, et al. Health Care Professionals' Experiences and Perspectives on Using Telehealth for Home-based Palliative Care: Scoping Review. J Med Internet Res 2023;25:e43429. [Crossref] [PubMed]
58. Borges do Nascimento IJ, Abdulazeem H, Vasanthan LT, et al. Barriers and facilitators to utilizing digital health technologies by healthcare professionals. NPJ Digit Med 2023;6:161. [Crossref] [PubMed]
59. Jarva E, Oikarinen A, Andersson J, et al. Healthcare professionals' perceptions of digital health competence: A qualitative descriptive study. Nurs Open 2022;9:1379-93. [Crossref] [PubMed]
60. Pereira L, Jerónimo C, Salgado A, et al. Patient centricity as strategy to improve quality of service in healthcare management. International Journal of Healthcare Technology and Management 2023;20:1-15. [Crossref]
61. Liang Z, Howard P. Professionalism and patient-centred care—patients’ views and experience. J Hosp Manag Health Policy 2023;7:19. [Crossref]