OJBM  Vol.7 No.1 , January 2019
A Framework of Output Specifications and Evaluation Method for Hospital PPP Projects
Author(s) Yun Pei
Introduction: Public-private partnership (PPP) has been adopted widely in infrastructure construction projects, and its application in medical and health service industry is increasing day by day. However, the performance of a PPP project could be affected by a number of factors and their interactions, due to its complexity in multiple stakeholders and stages. Previous researches on hospital PPP projects mainly focus on the factors which lead to success/failure, or the results of implementing PPP on hospital projects, or risk management, rather than its evaluation mechanism. So, to comprehensively promote and guarantee the effective implementation of the hospital PPP project, a scientif-ic quantitative evaluation method according to certain evaluation procedures and the performance evaluation system for hospital PPP projects are pro-posed from the perspective of the government.

1. Introduction

Public-private partnership (PPP) is cooperation between the public and private sector that combines the abilities of partners involved to achieve better outcomes, and is seen as a revolution in construction industry, being welcomed by agencies and governments. What’s more, PPP projects lay more emphasize on hospital assets, with all focus on leveraging capacities and integrating resources of both public and private sectors, just to introduce funding and expertise into healthcare sector.

To achieve the common goals, both the public and private sectors need to sign an agreement and form a project company to help construct the project. In PPP hospital projects, the public sector is in charge of the operation of core health services, clinical services for instance. Meanwhile, private sector is concentrating on no-core services, such as safety secure and cleaning. However, there are still problems that occur in such projects, caused by complexity of multiple stakeholders and long periods of construction. So, research mainly focused on the risk management of hospital PPP projects, and quantities of studies put effort into how to save costs. However, in some projects, the government’s specific efficiency gains from part of the successful bidder’s proposal evaluation, and it may be possible to include performance indicators related to the availability of these efficiency gains when the project is operational. According to literature review and investigations, there are few examples of PPPs in the hospital sector in developing countries, and how to comprehensively evaluate the performance of a hospital PPP project from the standpoint of the public sector has not been proposed yet. In order to deal with such problem, the project company shall regularly monitor and analyze the construction and operation of the project according to the concession agreement, conduct performance evaluation with the relevant departments, and establish a mechanism for adjusting the price or financial subsidies according to the performance evaluation results and in accordance with the concession agreement. The quality and efficiency of the public services will then be guaranteed. Project companies should also consider public opinions as an important part of monitoring the implementation of the hospital PPP project and performance evaluation.

Based on the above research status, this paper sets output specifications and constructs the framework of evaluation indicators for evaluation, using analytic hierarchy process (AHP) method to assess the implementation of hospital PPP projects. Finally, the framework and its evaluation method are applied in a specific hospital PPP project as a case study. Therefore, the result of study can be provided to the public sector to regulate and guarantee the successful implementation of hospital PPP projects, and to improve the level of management of medical and health.

2. Research Methodology

2.1. Questionnaire Survey

The study adopted a questionnaire survey approach using a self-administered questionnaire, released by mail. The mail survey approach was considered as feasible due to samples that are geographically dispersed. Respondents were enticed to responds with rewards. The survey was supported by personal face-to-face interview. The interviewers were guided by a set of similar questionnaire used in the mail survey. A total of 142 questionnaires were e-mailed and 69 questionnaires were collected in this survey, of which 58 were valid questionnaires. Among the effective responses, there are 13 experts, scholars in universities or academic institutions, 17 government officials, 28 design institutes, contractors and other business people. Most of the respondents have a rich PPP project (average 12 years, maximum duration of 38 years) experience. Additional 31 responses were obtained from the personal face-to-face interview. This technique will also help to reduce the rate of misinterpretation of questionnaires and to prevent respondent’s error.

The sample size of this questionnaire is relatively small, probably due to the following two reasons: Firstly, the application of PPP mode in China started late, there are not much experienced experts or scholars in this field as expected. Secondly, the questionnaire contains 22 primary indicators and 56 secondary indicators, which makes filling the questionnaire time-consuming, thus retard many respondents. However, the respondents of this questionnaire are targeted to the relevant personnel in the PPP fields within a certain scope, and the response rate is 40.8%. According to the Moser and Kalton research, when the questionnaire survey rate is less than 30% - 40%, the result of questionnaire survey is biased [1] . So the survey result is credible and effective in terms of response rate. For the reliability and validity of the questionnaire survey results, the paper uses Alpha reliability coefficient for reliability test and KMO value for validity test. According to SPSS analysis, Cronbach’s Alpha was 0.897 and the KMO value was 0.876, indicating that the consistency of the questionnaire was good and the questionnaire was valid. The identified indicators were stable and reliable.

In the questionnaire, the output specifications and indicators through literature review were listed, and questions were written to inquire suggestions on the reduction/increase of indicators by respondents. While interviewing, pairwise comparison of indicators was asked and recorded as data to estimate weights of each indicator. After the evaluation system was created, an interviewee was asked to provide a score of each indicator to get the whole evaluation score for the output of the hospital PPP project which is actually put into operation.

2.2. Determination of Weight Vector ω by AHP

The analytic hierarchy process (AHP) was officially proposed by the American operations researcher T. L. Saaty in the mid-1970s [2] . It is a systematic and hierarchical analysis method combining qualitative and quantitative methods. Due to its practicality and effectiveness in dealing with complex decision-making problems, it has quickly gained worldwide attention. Its applications span the areas of economic planning and management, energy policy and distribution, behavioral science, military command, transportation, agriculture, education, talent, health, and the environment.

The basic idea of AHP is basically the same as the process of thinking and judging a complex decision-making problem. This method provides a basis for quantitative evaluation of indicators, selection of optimal solutions, and has been widely used. Figure 1 shows the hierarchical structure of AHP model.

The objectives of the decision, the factors considered (decision criteria) and the decision objects are divided into the highest layer, called Objective (Level 1).

Figure 1. Hierarchy structure of performance evaluation indicators.

The middle layer concludes factors to consider, guidelines for decision making, namely Criteria (Level 2). The lowest layer contains the alternatives when making decisions, called Attributes (Level 3). Hence, the hierarchical structure diagram is drawn [3] .

The weight of each indicator is determined by means of pairwise comparisons of the activities to indicate the power with which one activity dominates another.

The pairwise comparison is established using a nine-point scale that converts the human preferences between available alternatives as equal importance, weak importance, strong importance, very strong importance and absolute importance, and each scale is empowered 1, 3, 5, 7, 9 relatively. As for Intermediate values between adjacent scale values, we use 2, 4, 6, 8 as transition scores. The comparison is based on experts’ judgment. Suppose n experts take part in the survey. Each expert’s opinion is obtained and analyzed individually to determine the weight vector ω i belonging to Indicator Group i (i = 1 − l). The following procedures provide a method to obtain the weight of indicators at Level 3 in Figure 1. As is shown in Figure 1, Indicator Group 1 is composed of x number of indicators, and is defined by { α 1 , α 1 , , α 1 }

Then, the pairwise judgment matrix for the weight of indicators based on the kth expert’s thoughts (k = 1 − n) is an x-by-x nonzero reciprocal matrix P, as presented below (Pij means the pairwise value after comparing αi with αj):

P = [ 1 p 12 p 1 x 1 / p 12 1 p 2 x 1 / p 1 x 1 p 2 x 1 ]

As an example, the eigenvector of indicator is calculated by

ω 1 k = ( ω 1 k 1 , ω 1 k 2 , , ω 1 k n )

ω 1 k i = j = 1 n p i j n

Then the eigenvector is normalized by

ω 1 k i = ω 1 k i j = 1 n ω 1 k j

Based on the scores given by Expert k, the weight vector ω 1 k of the indicators of Indicator Group 1 is: ω 1 k = ( ω 1 k 1 ω 1 k 2 ω 1 k n ) .

At last, a way to prevent artificial error and a consistence check need to be conducted until the precise result is obtained. The maximum eigenvalue λ max is a measure of consistency of judgement.

λ max = 1 x i = 1 x P ω 1 k ω 1 k i

CR = CI RI = 1 RI ( λ max 1 x 1 )

CR represents the consistency ratio; CI represents the consistency index; Ri represents average consistency index, as is shown in Table 1. When CR < 0.1, the matrix has satisfactory consistency; otherwise, it needs to be adjusted.

After all the results are calculated and the weight vector ω 1 k ( k = 1 m ) are worked out, the weight vector for Indicator Group 1 is determined by:

ω 1 = 1 n i = 1 n ω 1 k = ( k = 1 n ω 1 k 1 n k = 1 n ω 1 k 2 n k = 1 n ω 1 k x n )

The weight vectors ω 2 ω 2 l for indicators at Level 3 of Indicator Group 2-l can be obtained by means of the above method. And the weight ω m for the indicator groups at Level 2 can also be calculated with the method.

3. The Framework of Output Specifications for Hospital PPP Projects

According to the literature review and a questionnaire survey conducted previously, the output specifications and indicators can be synthesized and summarized into five categories, namely economic evaluation, operation & maintenance evaluation, medical technology evaluation, evaluation of hospital management level, public satisfaction evaluation [4] [5] [6] . These five categories will be discussed separately in the following paragraphs, and the weights are calculated using the data collected from questionnaire survey and method mentioned above.

3.1. Economic Evaluation

As is shown in Table 2, this section mainly evaluates the economic situation of hospital from perspectives of economic benefits of resource allocation, financial indicator, the operation cost of hospital, and medical price management level.

The economic benefits of resource allocation shall be assessed by the use of resources in terms of personnel, beds and equipment. It mainly investigates whether hospitals can organize production factors such as personnel, beds and equipment according to medical needs during operation, maximize economic benefits, and avoid waste of resources caused by idleness.

Table 1. Average random consistency index.

Table 2. Economic evaluation.

Financial indicators are based on the internal perspective of the hospital to consider the financial situation of the hospital, the main indicators are: total return on assets, net assets turnover rate, hospital operating efficiency rate, operating cost growth rate.

The operation cost of hospital means to assess the economic status of hospital management and decision-making from the aspects of direct cost and indirect cost, classifying, recording, collecting, and analyzing reports on various expenses in the medical service process, analyzing relevant costs information.

The level of medical price management, that is, from the perspective of the degree of openness and transparency of the price, the procurement process, and the true and accurate price information, whether the hospital’s drug price and the medical treatment price are controlled within a reasonable level, and truly serve the people’s livelihood.

3.2. Operation & Maintenance Evaluation

As is shown in Table 3, this section evaluates hospital operation and maintenance of hospital PPP projects from the perspective of third-party regulatory agencies [4] . When the social capital party participates in the construction and design of the hospital PPP project, it is mainly responsible for providing the repair and maintenance of the building during the operation period.

The evaluation of hospital operation and maintenance is divided into maintenance and management of construction equipment system and evaluation of logistics management work.

The maintenance and management of construction equipment system takes up 60% of “Operation & Maintenance Evaluation”, which makes it extremely important when a hospital PPP project is evaluated from this perspective. The hospital’s operation status is inspected from the perspective of hospital construction equipment hardware. The index is divided into construction equipment

Table 3. Operation & maintenance evaluation.

maintenance and repair system, maintenance technology advancement, maintenance timeliness, maintenance process impact on hospital operation, emergency Handling job evaluations.

The evaluation of logistics management is to investigate the operation of the hospital from the soft environment of daily affairs indirectly related to hospitals and medical treatment. The indicators are divided into logistics management system, parking lot, reasonable traffic organization, canteen hygiene, food safety, environmental sanitation, green landscape, Seven tertiary indicators for sewage discharge and solid waste disposal.

When evaluating the infrastructure of a hospital, special considerations should be given to the layout, function setting, medical procedures, safety requirements, sanitation requirements, environmental protection and supporting facilities of the hospital building; the climate and resources of the local area should be combined according to the principle of adapting to local conditions. So, the environmental, economic and cultural characteristics were evaluated in this paper.

Operational management evaluation is carried out during the operational management phase. The evaluation of the operation and management stage shall be carried out after the completion of the design scale and normal operation for more than one year (including one year), after the construction and equipment audit.

3.3. Medical Technology Evaluation

As is shown in Table 4, this section evaluates medical technology for hospital PPP projects from the perspective of third-party regulators.

The evaluation of medical technology is divided into service level of outpatient and inpatient, surgical and anesthesia skill, and quality of nursing.

Table 4. Medical technology evaluation.

Service level of outpatient and inpatient is divided into four aspects, including the qualified rate of prescription, admitting and discharge diagnosis accuracy, the success rate of critically ill patients rescue, and the average length of stay.

Surgical and anesthesia skill is divided into the incidence of complications of the patients and the mortality of anesthesia, that is, accidents caused by improper operation of anesthesia during surgery.

Quality of nursing is evaluated from the perspectives of medical care ratio, guard bed ratio, nurse-to-patient ratio, implementation of job responsibilities, and number of patient care accidents. Different from the above grading, quality of nursing is assessed comprehensively, so secondary indicator doesn’t show up in Table 4.

3.4. Evaluation of Hospital Management Level

As is shown in Table 5, this section evaluates hospital management for hospital PPP projects from the perspective of third-party regulators.

The evaluation of hospital management is divided into volume of medical service, level of medical service, evaluation of medical environment, level of medical information management, mandated task by the government, doctor-patient relationship and patient safety.

Volume of medical service is divided into three aspects: patients at emergency department, hospitalized patients discharged, and number of people who did surgery.

Level of medical service is divided into the reserved medical service level, waiting time at the service window, basic medical security service management level, clinical pathway management and electronic medical record establishment.

Level of medical information management is assessed from the following perspectives, namely surveillance system and intelligent management information system.

Mandated task by the government is evaluated from counterpart support, health education and public health emergency rescue.

Table 5. Evaluation of hospital management level.

Patient safety is evaluated from three aspects, medical safety, nursery safety and medication safety respectively.

Evaluation of medical environment and doctor-patient relationship are evaluated according to the average score of different aspects.

3.5. Public Satisfaction Rating

In this section, the hospital PPP project is evaluated from the perspective of Public satisfaction rating, indicators and weights to assess were described in Table 6. This evaluation standard is based on the satisfaction assessment of the public PPP medical project, and is entrusted by a third-party regulatory agency by the supervisory function in the implementation of the PPP project.

Public satisfaction can measure the extent to which medical development of medical PPP programs provides social infrastructure and transportation services to meet user expectations. The evaluation results of public satisfaction can provide important decision support for the improvement of medical management level of PPP project operators and improvement of medical service level.

Table 6. Public satisfaction rating.

Public participation is a necessary procedure for the evaluation of PPP projects. Public participation means that the legal interests of the public, groups, and units, in which the project is located are fully guaranteed in order to make the argumentation of the PPP construction project more scientific and reasonable.

The purpose of public participation is to consider and compensate the interests and opinions of the groups directly or indirectly affected by the project through effective consultation with the public. Accepting social supervision and encouraging the public to participate in the supervision of the project is to effectively improve the implementation level and efficiency of the project. It is also an effective way to reduce the problems that may arise from the project construction and improve the social and environmental benefits of the construction project.

The government shall set up special funds for investigations for the public satisfaction rate survey, and the PPP project operators can actively cooperate and conduct public satisfaction surveys in accordance with the requirements of the Ministry of Health and the Ministry of Environmental Protection and survey specifications.

The evaluation method uses a third-party regulatory agency to issue a public opinion questionnaire or a household visit to the affected groups of the PPP project, conduct statistical analysis on the opinions of the respondents, and perform verification scores based on the questionnaire survey.

The third-party regulatory agencies of the PPP project are required to collect public opinions and, after verification, provide feedback to the government and PPP project operators.

The PPP project operation unit announces the performance evaluation status of PPP medical projects to the public every year; and can actively cooperate with the establishment of local PPP project supervision, complaints, and reporting hotlines, and there are special people to answer; the complaint handling rate is 100%.

Based on the above research and results, the final performance evaluation of hospital PPP project can be summarized in Table 7.

3.6. Application

Using the data of a certain PPP hospital provided by an interviewee charged of operation, it is estimated that the total score for the hospital PPP project which has been put into operation for 2 years in the UK equals 86.78 (100 is full mark).

Background information provided by the interviewee was as follows:

The project was constructed to improve the network of urban clinics and provide publicly funded health care services with hospitals. The PPP hospital project includes the construction of a new 489-bed national referral hospital, a portal clinic near the hospital, and refurbishment and re-equipment clinics. The second purpose of PPP is to engage the private sector in new ways to ensure that these facilities operate effectively and to provide more effective, higher quality care and a wider range of services. The English PPP hospital project is managed by government employees whose operations are governed by laws, regulations and traditions that govern public sector procurement, human resource management, finance and administration. Partnership mechanisms, including a comprehensive

Table 7. Performance evaluation of hospital PPP project.

network of contracting construction and operations, and the use of output-based payment mechanisms, aim to increase accountability for service delivery and quality of care by reducing government accountability for day-to-day operations.

Through tendering, the alliance of social capital companies signed contract, agreed to build, operate, manage and provide clinical and non-clinical services through the integrated network over 20 years. The project has a total value of about $120 million. Capital expenditures are co-financed through public (42%) and private funds 58%).

Based on the above information, a whole new questionnaire was designed to get an overall rating of the PPP hospital project. Firstly, when the investigating the stakeholders of this PPP project, scores on each dimension listed in the above tables were collected, and then we use SPSS to confirm the validity and consistency of the data and remove the deviated number from database. 48 sets of data were proved valid. Then, after calculating the average number of each category, the weight provided by previous study was used to be multiplied by the average score of each corresponding indicator. Finally, the overall rating of the hospital PPP project was calculated (87.5), which means there’s still much room for improvement.

4. Conclusion

The study elucidates into the output specifications of hospital PPP projects and works out a way to quantify the indicators of performance evaluation system. At last, a case study was conducted to further confirm the validity and feasibility of the evaluation model. However, the weight of each indicator might not suit every hospital PPP project, and the indicators should be updated as project changes. What’s more, more scientific methods should be put forward to further eliminate subjective bias when determining weights of indicators, for example, fuzzy analytic hierarchy process (FAHP) approach [7] , grey correlation analysis method and so on.

Cite this paper
Pei, Y. (2019) A Framework of Output Specifications and Evaluation Method for Hospital PPP Projects. Open Journal of Business and Management, 7, 167-179. doi: 10.4236/ojbm.2019.71012.
[1]   Wijewardena, H. and Zoysa, A.D. (2005) A Factor Analytic Study of the Determinants of Success in Manufacturing SMEs. Faculty of Commerce—Papers.

[2]   Saaty, T.L. (1967) Modern Nonlinear Equations. Dover Publications.

[3]   Saaty, T.L. and Vargas, L.G. (2012) Models, Methods, Concepts & Applications of the Analytic Hierarchy Process. Springer US.

[4]   Vian, T., Mcintosh, N., Grabowski, A., et al. (2015) Hospital Public-Private Partnerships in Low Resource Settings: Perceptions of How the Lesotho PPP Transformed Management Systems and Performance. Health Systems & Reform, 1, 155-166.

[5]   Chen, M., Zhu, Q.C. and Dong, D.M. (2011) Game Study on Construction Safety Supervision Based on System Dynamics. China Safety Science Journal, 21, 73-81.

[6]   Javed, A.A., Lam, P.T.I. and Chan, A.P.C. (2013) A Model Framework of Output Specifications for Hospital PPP/PFI Projects. Facilities, 31, 610-633.

[7]   Carr, V. and Tah, J.H.M. (2001) A Fuzzy Approach to Construction Project Risk Assessment and Analysis: Construction Project Risk Management System. Advances in Engineering Software, 32, 847-857.