Introduction of Chinese hospital ranking method from the aspect of theoretical framework, realistic choice and social effect
Introduction
Due to the nature of medicine as a specialized field, there exists an obvious imbalance in medical knowledge between doctors and patients. It is vital that a hospital can provide the most appropriate medical services when people are in severe conditions. However, the patient may not know which hospital would best suit their needs as they lack access to relevant information. Although advertisements and general practitioners are in the position to recommend certain hospitals, that information is biased and maybe result in a wrong decision on the patient’s part, potentially resulting in delayed diagnosis and treatment of the disease.
However, an authoritative list of hospitals can help change this situation. Since 1990, U.S. news and world reports (1) have made quality assessments of U.S. hospitals, providing a ranking of “the best hospitals”. From 1993 to 2004, the method was completed by the National Center for Public Opinion Research (NORC) at the University of Chicago. The assessment after 2005 was completed by RTI International of the North Carolina Triangle Research Park. The U.S. hospital rankings play a positive role in guiding patients to seek medical care in the United States (2), while providing a more objective database for comparison between hospitals (3).
In view of this, the Hospital Management Institute of Fudan University decided to create a list of “the best hospitals in China” to help patients in their search for a hospital. Since 2010, the Hospital Management Research Institute of Fudan University has published the list annually, which ranks the top 100 hospitals in China and the top 10 hospitals over each of the 37 clinical specialties in China. After 7 years of inspection, the ranking published by the Hospital Management Institute of Fudan University has been accepted by most hospitals, and has gradually become one of the most important hospital rankings in China, serving as a form of guideline to patients when seeking treatments.
The methodology in creating the list of Chinese hospitals was based on the US’ method of obtaining their list of “the best hospitals”, but has been modified in many ways to meet with the conditions and requirements in China, placing emphasis on a hospital’s reputation and its capacity to conduct scientific research (4-6). In this paper, we will introduce the unique method used for the Hospital Management Institute rankings in China and share our experience from the aspects of theoretical framework, practical choice and practical effect.
Theoretical framework
We believe that China’s hospital list should play its role in two ways. One, to reflect the level of clinical specialties and reputation among China’s hospitals objectively and fairly, which will guide patients to treatment of severe diseases. Two, to set the benchmark for hospitals, which will help in ensuring medical development.
The quality of medical services provided by hospitals can be measured in three ways: infrastructure, service process and service results (Figure 1). Excellent medical service begins with hospital infrastructure, and is reflected in the patient’s diagnosis and treatment. The combination of infrastructure and diagnosis determines the patient’s results, which need to be adjusted to fulfill the complexity and risk of the patient’s disease. This is so that the diagnosis and treatment results can be compared in different hospitals or clinical specialties.
Evaluation criteria for infrastructure should differ depending on the characteristics of their respective clinical specialty. The items for evaluation generally include: the number of hospital beds, medical equipment, and other environmental characteristics. In the United States, the capacity data can be collected from the Medicare Provider Analysis and Review (MEDPAR) database maintained by the federal Medicare and Centers for Medicaid Services Center (CMS) (7). The database includes information provided by all Medicare beneficiaries who have received hospitalization services. Other data relating to technology, beds, instruments and equipment are mainly derived from the annual survey of the American Hospital Association. In China, data such as the number of beds, the number of instruments and equipment may be obtained from the Statistical Center of the Ministry of Health (8). However, the database is still in the process of construction. As the data is based on information reported by the hospitals themselves, a direct usage of the provided data will result in a distortion of the statistics. Furthermore, adding indicators that rank hospitals according to their size may propel public hospitals to prioritize large-scale development, which may go against the development of service efficiency and quality. With these considerations in mind, we decided to not include the number of beds, instruments and equipment in our evaluation. Research ability as a soft infrastructure, however, is regarded as a factor equally as important as a hospital’s hard infrastructure. SCI-published papers and national science and technology awards are frequent indicators reflecting a hospital’s research ability. In addition, the number of SCI-published papers can be attained from the SCI citation database, and national science and technology awards can be certified from the national science and technology awards announcement (9), (which are significantly less biased as opposed to hospital reports). Therefore, when ranking Chinese hospitals, these two indexes are used to reflect the research ability of hospital specialists, and inadvertently, reflect the sustainable development of the hospitals.
Similarly, the evaluation criteria for service result generally includes outpatient visits, inpatient visits, emergency visits, patients undergoing surgery, case fatality rate, safety accident rate and so on. All these indicators rely on accurate data collection, which may have been more easily achieved in the United States, but faces some difficulties in China. We also propose that adding workload-related indicators to the evaluation will aggravate the situation of supplier-induced demand. Therefore, ‘service result’ is not included in the methodology.
The grading of the service process is primarily aimed at the evaluation of the reputation of specialists in the hospitals, which is a reflection of the hospital’s reputation of providing quality service within a certain specialty. In the United Stated, the practice of creating a hospital ranking is sending questionnaires randomly to doctors registered in the physician archives of the American Medical Association, asking them to nominate five best hospitals in specific clinical specialties without considering the price and location. The number of doctors surveyed has increased year by year, coming up to 3,400 in 2005. The number of doctors surveyed in each specialty is approximately 200. The premise is that doctors in the United States are trained in large hospitals and are familiar with major hospital specialties. However, in China, due to the lack of a unified training platform, doctors differ in terms of their basics, and the difference between practicing physicians in township hospitals and those of tertiary hospitals is obvious. If the average doctor is selected as the object of investigation, the survey would lack in reliability. In this regard, we decided to select the doctors from members of the Chinese Medical Association and the provincial Medical Association’s chairmen and vice-chairmen.
In conclusion, from an operational perspective, indicators such as the number of patient, the number of beds, the type of equipment have not been included in the evaluation scope. Instead, social reputation is used to reflect “process” and the ability to develop sustainably is used to reflect “structure”, and in combination, these two dimensions determine the Chinese hospital rankings (Figure 2). Through expert consultation, in order to highlight the evaluation of social reputation, the weight of social reputation is set as 0.8, and the weight of the sustainable development ability (SDA) is set as 0.2.
Methods
Valuating the social reputation
The process of evaluating the social reputation of clinical specialist in hospitals is as follows.
(I) Determine the clinical specialty list
There are a total of 37 clinical specialties: pathology, infectious department, otolaryngology, radiology, respiratory, rheumatology, gynecology and obstetrics, orthopedics, psychiatry, dental, anesthesia, urology, endocrinology, dermatology, general surgery, neurology, kidney disease and nerve surgery, digestive disease, pediatric medicine, pediatric surgery, cardiovascular disease, cardiac surgery, thoracic surgery, hematology, ophthalmology, cosmetic surgery, oncology, geriatric medicine, rehabilitation medicine, medical laboratory, burns department, nuclear medicine, ultrasonic medicine. This list will be updated along with the development in the various clinical specialties in China.
(II) Determine the subject of investigation
In year 2016, a total of 4,173 people participated in the survey, including the chairman and all members of the Chinese Medical Association, the chairman of the provincial Medical Association, and the vice-chairman of the junior commission of the Chinese Medical Association to increase the reliability of the survey. The list of respondents was adjusted annually according to the change in membership.
(III) Preparation of questionnaires
The research method is questionnaire. Each specialist needs to fill out 10 hospital names on the questionnaire (Figure 3).
(IV) Send the questionnaire
Send the questionnaire by registered mail. A letter of thanks, envelope and return postage stamps are also placed in the registered letter. At the same time, an email with the same content is sent to the experts to remind them to fill out the questionnaire and sign them.
(V) Recovery of questionnaires
After two weeks of sending out the questionnaire, the task group will arrange people to call and email the unanswered experts and remind them to send back the questionnaires. In the third week of sending out the questionnaire, the task group will again arrange people to call the experts who did not respond, and try to improve the recovery rate of the questionnaire. In 2016, a total of 2,657 experts responded to the survey, which reached 63.7 percent, which is well above the 62.7 percent recovery rate in 2015.
(VI) Enter the questionnaire to form a database
The questionnaire was recorded by three individuals simultaneously, so that input error can be recognized in time, which ensures the consistency between the input data and the original data. Excel software is used to import data, and the data lock function of excel software is used to ensure that the data could not be modified. The function of software is used to link the input data and analysis results.
(VII) Calculate the ‘specialty reputation score’
The score given by an expert to a certain hospital will be calculated and further interpreted as the ‘special reputation score’ for that hospital. Considering that the difference between top hospitals is not very significant, in order to widen the gap between the top hospitals, especially for the first top hospital to stand out, we have designed the questionnaire in a way that if the expert nominates a hospital in the first place, then the hospital gets 12 points; second place is 10; third place is 9; fourth place is 8; fifth place is 7; sixth place is 6; seventh place is 5; eighth place is 4; ninth place is 3; tenth place is 2. In addition, the highest and the second highest scores will be taken out in the process of calculation to guarantee fairness of evaluation, as experts may possibly be biased towards the hospital they are working for. This means that a hospital must at least have 3 nominations from 3 experts to get a chance to appear in the final hospital ranking list. The score of the specialist nomination can then be added as the score of specialist reputation of a hospital.
For example, in all 51 neurology experts, 35 experts nominated Hospital A, 20 experts of which nominated Hospital A as first, 6 experts nominated Hospital A as second, 5 experts nominated Hospital A as third, 3 experts nominated Hospital A as fourth, 1 expert nominated Hospital A as fifth. Therefore, Hospital A’s neurology specialty score can be calculated as: (20−2)×12+6×10+5×9+3×8+1×7=352.
The formulation is as such:
SR is the specialty reputation score. ∑SN is the valid sum of specialist nomination score. N is the number of experts in poll.
In practice, for each clinical specialty, the ten hospitals with the highest SR scores will be highlighted as Top Hospitals in the hospital reputation rankings (HR rankings), and the rest hospitals with ∑SN >5 will be listed as nominating hospitals.
(VIII) Standardize specialty reputation score
In order to make the difference of ‘specialty reputation score’ more comparable between hospitals, we standardized the hospital specialist reputation score of each hospital.
The formulation is as such:
Standardized specialty reputation (SSR) is the SSR score. SR is the specialty reputation score. MAX(SR) is the maximum value of specialty reputation score. MIN(SR) is the minimum value of specialty reputation score.
(VIIII) Calculate the ‘HR score’
Summarize all SSR score of a hospital to obtain its reputation score. By putting together, a list of hospitals according to their hospital reputation score in descending order, the hospital reputation rankings are obtained.
Valuating the ability to develop sustainably
Through the SCI citation database and the national science and technology reward announcement, we can obtain data on every hospital’s SCI paper publish count and the number of national science and technology reward awarded to them. The data is objective, accurate, and reflects the hospital’s scientific research ability, and in turn, reflects the sustainable development of the hospital. Therefore, we select the number of SCI papers, the number of national science and technology rewards to evaluate the sustainable development of each hospital. The specific method is:
(I) Obtain data on SCI papers and national science and technology awards
Choosing top 150 hospitals from reputation rankings, authorize the Medical Institute of Technical Information to retrieve SCI papers, journal impact factor, as well as the national science and technology award. The retrieval time was set as a year between January 1 - December 31. In order for an SCI paper to be included in a hospital’s SCI paper count, the hospital should be listed as the first author’s affiliation. On the other hand, 3 kinds of awards among the national science and technology award are included in the evaluation, namely the National Prize for Natural Sciences, the National Prize for Progress in Science and Technology, and the National Award for Technological Invention (10).
(II) Calculate the research output score of each hospital
Summarize the influence factors of the published SCI papers of each hospital to get the SCI score. A first prize national science and technology award is allocated a score of 100, and a second prize is allocated 50. The sum of SCI and award scores are considered the research output score to represent the sustainable development capacity of hospitals.
ROS is the research output score. SSCI is the score of SCI. SA is the awards score.
(III) Standardize the “SDA score”
In order to make the difference of sustainable development ability score between hospitals to be more comparable, we standardize the sustainable development ability score of each hospital.
The formulation is as such:
SDA is the sustainable development ability score. MIN(ROS) is the minimum value of research output score. MAX(ROS) is the maximum value of research output score.
Valuating comprehensive score
(I) Calculate the comprehensive score of hospitals
In order to combine the social reputation score and the sustainable development ability score, after several rounds of expert advice, we decided to allot a proportion of 80% for the social reputation score and 20% for the sustainable development ability score.
The formulation is as such:
CS is the comprehensive score of hospitals. HR is the hospital reputation score. SDA is the sustainable development ability score.
(II) Obtain hospital comprehensive ranking
According to the comprehensive score of hospitals, the hospital comprehensive ranking is created and made public online (11).
The form of hospital rankings
By following these operations step by step, the methodology of the hospital ranking of Hospital Management Institute, Fudan University is formed, and the result is calculated and published yearly. The final result of this methodology concludes four different hospital ranking lists. The first one is ‘the hospital reputation rankings of each clinical specialty’, which is obtained up till step 7 of valuating the social reputation. The second one is ‘the hospital reputation rankings’, which is obtained up till step 9 of valuating the social reputation. The third one is ‘the hospital comprehensive rankings’, which is obtained once the final step has been completed.
In our calculation, we have also segregated China into seven regions: north China, central China, south China, east China, northwest, northeast, southwest, and published the top 20 comprehensive hospitals of each region as ‘the regional hospital comprehensive rankings’. The regional hospital rankings aim to help patients satisfy their health demand by sorting out hospitals that are relatively closer to them, which can also avoid large numbers of patients crowding into big cities.
Results
Till today, the object hospitals are all hospitals in mainland China, and the hospital ranking list has been published for the past 7 years. Every year, we issued the 4 kinds of ranking lists online. With many years of development, the Chinese hospital ranking list released by the Hospital Management Institute, Fudan University, has become to be one of the most influential hospital ranking list (12). The detailed context is made available on the website of Hospital Management Institute, Fudan University (13).
Notably, since 2009, this methodology has been constantly improved and the results are more stable. Its stability is seen in the following aspects: firstly, the top 10 hospitals from ‘the hospital reputation ranking’ have been relative stable with few changes. The top 4 hospitals’ ranking, in particular, have been unmoved, and the following ranking hospitals have had few changes over the 7 years. Secondly, while the ‘hospital comprehensive rankings’ has relatively changed, the ranking rangeability of the top 100 hospitals in the list remains relatively stable over these years.
The degree of physician participation has seen outstanding progress in the past years. The number of clinical specialties evaluated has increased from the original 30 to 37, the number of experts who filled out the questionnaire increased from 1,824 to 4,173, and the questionnaire response rates have been increased from the original 44.24% to 63.67%. For each clinical specialty, the questionnaire response rates are all greater than 50%, and the highest questionnaire response rates among all clinical specialists is nuclear medicine at 81.6%.
With the development of Chinese hospital rankings produced by the Hospital Management Institute, its credibility is constantly growing.
Discussion
It is without doubt that the Chinese hospital rankings are successful even though it includes no quantitative analysis, because it brings about great social effects in many ways.
In this part, we will discuss several key factors that promote the success of it. Firstly, it includes all top physicians from each clinical specialty with high response rate, which increases the credibility of the results of assessment. Secondly, the methodology of this hospital ranking is very simple. It relies on the result of peer review and scientific research output, and the questionnaire sent to the experts is made up of several simple questions, which guarantees the operability of the ranking work. Thirdly, the response rate of physician participation in the questionnaire was 44% in 2009. Up till 2016, the answer rate has already increased to 63.67% and shows an increasing trend. Fourthly, it is purely for the public good. The institute ranking the hospital list charges nothing from hospitals, and has created high social benefit in China. For hospitals, it helps establish a benchmark of clinical specialties, and promotes the development of hospital medical specialties. For the public, it helps decrease the degree of imbalance in information during one’s search for health and treatment. The success of the hospital ranking list produced by the Hospital Management Institute takes a lot of will and perseverance. Without these efforts, it would not have been able to obtain great reception from the public.
In this part, we will discuss the great social effects that it brings. First, for patients, it will provide some useful information about hospitals when they contract severe diseases. Second, for hospitals, it helps to set the benchmark, which will help medical specialized development. Third, it is used as reference of unit performance assessment within a hospital. Fourth, for the local governments, it serves as a guide to the disciplinary construction of medicine in the region.
At present, the evaluation process includes two main indexes: reputation evaluation index and research evaluation index. Due to the fact that the quality of data varies from place to place, it is difficult to use the data to evaluate hospitals. Therefore, quantitative analysis was not included. However, as the ability to collect data is perfected over the years, and the connectivity of the entire nation’s health data can be achieved, such information will be further supplemented into assessment.
Acknowledgments
Funding: Supported by the National Natural Science Foundation of China (No. 71473045), Supported by Key Projects of Philosophy and Social Sciences Research, Ministry of Education (No. 15JZD029).
Footnote
Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jhmhp.2017.11.01). The authors have no conflicts of interest to declare.
Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.
References
- Available online: https://health.usnews.com/best-hospitals
- Available online: https://health.usnews.com/health-care/best-hospitals/articles/faq-how-and-why-we-rank-and-rate-hospitals
- Guo S, Dong S, Liang M, et al. Comparative Analysis and Enlightenment of Three Major America’s Hospital Rankings. Chinese Health Quality Management 2014;(3):123-5.
- Wang J. The logic of Chinese hospital ranking, Fudan University. China Hospital CEO 2016;(6):64-7.
- Wang Z. The publish of Chinese hospital ranking, Fudan University. China Hospital CEO 2011;(1):23-8.
- Dong S, Guo S, He L, et al. Study of Present Situation and Countermeasures of China’s Hospital Rankings. Chinese Hospital Management 2015;35:38-40.
- Available online: https://www.cms.gov/Research-Statistics-Data-and-Systems/Research-Statistics-Data-and-Systems.html
- Available online: http://www.nhfpc.gov.cn/mohwsbwstjxxzx/
- Available online: http://www.nosta.gov.cn/web/index.aspx
- Available online: http://www.nosta.gov.cn/web/list.aspx?menuID=7
- Available online: http://www.fudanmed.com/institute/news2015-2.aspx
- Wang Z. The Chinese hospital ranking, Fudan University is working better. China Hospital CEO 2012;(22):54-5.
- Available online: http://www.fudanmed.com/institute/news222.aspx
Cite this article as: Xiong X, Gao J, Zhang B, Lu J, Luo L. Introduction of Chinese hospital ranking method from the aspect of theoretical framework, realistic choice and social effect. J Hosp Manag Health Policy 2017;1:4.
