Abstract
Background/purpose
Nowadays, Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score II (SAPS II) scoring systems have drawn much attention for the evaluation and prediction of disease process in patients admitted to intensive care units (ICUs). To use these scoring tools, their predicting power must be initially validated for the target patients. This study was conducted to evaluate the performance of these two scoring systems in an ICU for respiratory diseases in Iran.
Material and methods
All records of patients admitted during a 1-year period were retrospectively reviewed, and the APACHE II and SAPS II scores were calculated accordingly. Information gathering was performed using a questionnaire.
Results
A total of 415 records were used. The mean age of patients was 49.28 ± 0.94 years. Using receiver operating-characteristic curve, cutoff points for 80% sensitivity and specificity of mortality prediction for APACHE and SAPS scores were 13.5 and 27.5, respectively. Calibration and discrimination studies indicated an acceptable status for both scales, but APACHE II scoring system seemed to show rewarding outcomes.
Conclusion
Results indicate that APACHE II scoring system can be considered as a reliable method for predicting mortality in our referral respiratory ICU.
Keywords
APACHE: II; intensive care units; respiratory tract diseases;
1. Introduction
Although new advances in the treatment and better settlement of severely ill patients in the intensive care unit (ICU) have resulted in their survival, such measures prolong their ICU stay and increase hospital expenses. Patients being informed of the severity of illness at the time of ICU admission helps decide continuation of expensive treatment and avoids unnecessary procedures.
Use of scoring systems especially developed for patient assessment at the time of ICU admission has reduced many problems and facilitated treatment planning. On the other hand, these tools help evaluate and compare both quality and quantity of care between healthcare institutes.1, 2 Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score II (SAPS II)scoring systems are the two tools that are widely used by most ICUs to predict clinical outcome, which have been evaluated and validated by many centers and can be adjusted according to the needs.3, 4
The present study was designed to evaluate the performance of APACHE II and SAPS II scoring systems in predicting the severity of illness and the mortality of patients with respiratory diseases who were admitted to an ICU in Iran.
2. Materials and methods
This historical cohort study was performed using the medical records or documents of all patients who were admitted to our respiratory ICU, the most important referral ICU for respiratory diseases in the country, during a 1-year period. APACHE version II and SAPS version II scores were calculated using patient information at admission. Data gathering was performed using a specially prepared questionnaire.
2.1. Statistical analysis
Descriptive statistics were analyzed using mean ± SD for quantitative and frequency (percent) for qualitative variables. Initially, normal distribution of variables was assessed by Kolmogrov–Smirnov test, and nonparametric Mann–Whitney test was used to compare APACHE and SAPS results among the alive and dead patients.
The models' ability to distinguish patients who died in hospital from those who still survive is because of their discrimination power, which was assessed using receiver operating-characteristic (ROC) curve and area under the curve (AUC) as a composite index of discrimination. This curve was plotted for the models using the sensitivity and specificity values for true prediction of hospital death across a range of possible cutoffs of predicted mortality. The model is considered perfect if AUC reaches 1.0; subsequently, the best cutoff point for each score is calculated using the Youden value.
The degree of correspondence between the estimated probabilities of mortality produced by a model and the actual hospital mortality is the calibration of the model, which was assessed using Hosmer–Lemeshow “c” goodness-of-fit statistics. Initially, patients were rank-ordered according to their probability of death and were divided into 10 groups of approximately equal number of observations; the numbers of the expected and observed mortality were compared in these groups using Pearson statistics (c).
The model with the least c and the highest p value shows the best agreement between the observed and the expected number of deaths.
To calculate the standardized mortality ratio (SMR), the observed mortality was divided by the predicted mortality. Also, logistic regression was used to reach a model for prediction of death.
All statistical analyses were performed using SPSS version 16.
3. Results
3.1. Descriptive analysis
A total of 415 patients were enrolled into the study. The mean age of patients was 49.28 ± 0.94 years, with normal distribution as shown with Kolmogrov–Smirnov test and p > 0.05.Gender study showed that 60.6% of patients were male. The reasons for ICU admission were postintubation tracheal stenosis (20.4%), chronic obstructive pulmonary disease (COPD) acute exacerbation (3.9%), postoperation esophageal cancer (3.9%), bronchiectasis (3.6%), postop thyroid cancer (3.6%), and various other causes such as empyema, trachea-esophageal fistula, alveolar proteinosis, etc. In a total of 415 patients, cardiopulmonary resuscitation was performed for 21 (5.1%) and 75 (18.1%) eventually died (Table 1).
3.1.1. Correlation between acute death and APACHE II and SAPS II scores
To study the correlation between actual number of deaths and results of APACHE II and SAPS II scores, initially normal distribution of the two scores was evaluated using the Kolmogrov–Smirnov test. Neither of the two scoring results had normal distribution (p < 0.001); therefore, nonparametric Mann–Whitney test was used to compare APACHE and SAPS results among the alive and dead patients.
Results showed that the scores of the dead patients were significantly higher than that of the living patients. That is, APACHE score was, on average, 23.14 ± 8.19 in the dead group and 9.82 ± 6.19 in the living group (p < 0.001), and the SAPS score was 19.19 ± 13.33 and 46.14 ± 17.73 in the living and dead patients, respectively (p < 0.001) (Table 2).
3.1.2. Evaluation of prediction values of APACHE and SAPS scoring systems
Area under ROC curve was calculated to evaluate the prediction value of the scoring systems. For APACHE and SAPS scores, the areas were 0.897 (CI 95%: 0.858–0.937) and 0.887 (CI 95%: 0.847–0.926), respectively (Fig. 1). Youden value (sensitivity + specificity – 1) was calculated to reach the best cutoff points for both scores. The point with best Youden value was considered as the cutoff point. For APACHE score, the best cutoff point chosen was 13.5, with 90% sensitivity and 75% specificity, and a Youden value of 0.65. For SAPS score, it was 86.5, with 83% sensitivity, 77% specificity, and a Youden value of 0.60.
Download full-size image
Hosmer–Lemeshow goodness-of-fit testing was used to calibrate both scoring systems (Table 3, Table 4). Results showed better calibration of APACHE II score.CHLSAPS=7014p=0.522CHLAPACHE=3.27p=0.916
CHL SAPS ¼ 7014 p ¼ 0:522
CHL APACHE ¼ 3:27 p ¼ 0:916
SMR was calculated to compare the observed and expected mortality in both scores. Results are as follows:
SMR APACHE : observed=expected ¼ 0:183=0:2025 ¼ 0:903
SMR SAPS : observed=expected ¼ 0:183=0:8483 ¼ 0:215
As evident from these results, SMR for APACHE was closer to reality.
3.1.3. Logistic regression
In order to calculate the risk of death using APACHE and SAPS
scores, logistic regression was used to describe a model for each
scoring system. In this model, the response variable was one of two
optional variables (living or death) and APACHE/SAPS scores were
entered as predicting variables. The following two models were
used for calculating the risk of death:
For APACHE score : logitðpÞ
¼ 5:102 þ ð0:227 APACHE scoreÞ
For SAPS score : logitðpÞ ¼ 4:424 þ ð0:095 SAPS scoreÞ
The odds ratio for APACHE score was 1.255 (p < 0.001), which
means that 1 point increase in APACHE scores will increase the
odds of death by 25.5%. For SAPS score, this ratio was 1.099
(p < 0.001), indicating that each point increase in the score
increases the odds of death by 9.9%.
4. Discussion
Patient assessment at the initiation of ICU admission is important for mortality and morbidity prediction and ICU performance evaluation. In this study, we evaluated the two most popular scoring systems, APACHE II and SAPS II, in our referral ICU for respiratory diseases. The results showed that APACHE II score could predict patient outcomes more precisely than SAPS II score and, therefore, is considered to be more suitable for use in our respiratory ICU.
Illness severity assessment using scoring systems has become popular, especially for ICU patients. Because the incidences of mortality and morbidity are relatively high in ICU, these scores help predict patient outcomes, evaluate resource allocation, carry out triage of patients, and assess ICU ranking.5 Various scoring systems have been introduced, some being feasible and some complicated but reliable. The two most popular systems introduced to date are APACHE and SAPS, which are based on multiple logistic regression equations describing abnormalities of physiologic variables during the first 24 hours of ICU admission.3, 6
These scores have been evaluated in a wide range of patient populations and in many specialty- and subspecialty-related ICUs.7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 However, because studies performed in respiratory diseases setting are scarce and because the ICU of our hospital is solely dedicated to patients with respiratory problems (medical and surgical), evaluating the prediction power of these scores became the goal of this study.
Results of this study revealed that 13.5 and 27.5 could be considered as acceptable cutoff points for APACHE II and SAPS II, respectively, which could predict survival or death with a considerable sensitivity and specificity. A similar study conducted by Gupta et al1 in a respiratory ICU in India reported 12.87 ± 8.25 for APACHE II to be acceptable; in the same study, it was mentioned that similar scores had been reported by centers from Saudi Arabia and Singapore. Deša et al20 from Croatia have reported a score of 20 for SAPS II from a general ICU. Both of the above-mentioned studies have reported the prediction value of the scoring systems to be acceptable,with only minimal underprediction.1, 20
This seems very interesting because, as indicated, a great number of our patients were surgical cases spending their recovery period post operation and also we had medical cases with COPD exacerbation and long duration of ICU stay. This means that although we had a combination of short- and long-term admissions and also a combination of surgical and medical cases, we nonetheless reached the same conclusion as other centers that these scoring systems are well qualified for patients admitted to respiratory ICUs. The only fact that must be taken into consideration is that, since our center is a referral center, mostly high-risk and end-stage cases are admitted. Therefore, as indicated in the results, the mean value of the two scoring systems may be somewhat higher than expected, and therefore the results may be preferably generalized to referral centers such as ours.
To better clarify the situation it must be mentioned that when selecting the best severity of illness scoring system for ICU, precise information on discrimination, calibration, or goodness of fit must be gathered. Therefore, to evaluate the discriminative power of APACHE II and SAPS II, ROC curve and AUC must be calculated. As mentioned, these areas were 0.897 and 0.887 for APACHE and SAPS, respectively, which means that these two scoring systems had quite similar AUC and highly acceptable discrimination power for respiratory patients in our ICU. Other studies have also evaluated the performance of these two scoring systems in patients with respiratory diseases. For better comparison, the results of a study performed by Aggarwal et al21 from India can be mentioned. In this study, AUC values for APACHE II and SAPS II were reported to be 0.713 and 0.781, respectively. This study showed better results for these scores compared with similar studies in Taiwan (AUC = 0.686) and Italy (AUC = 0.809) for APACHE II and for SAPS II (AUC = 0.735).1, 22, 23
The main objective for using scoring systems at the beginning of ICU admission is to predict the mortality of a patient and, therefore, evaluate the management capabilities of an ICU; therefore, it is always required to calibrate any scoring system before making it a routine. Hosmer–Lemeshow test is usually performed to assess the calibration of scoring systems, and our results revealed better calibration of APACHE II score for our specific group of patients (p > 0.86).3, 20, 24, 25 We observed an SMR value of 0.9 for APACHE II in our study, which also indicated that APACHE II prediction was more realistic. Interpretation of the above-mentioned statistical analyses has been variable among different studies and with different researchers.20, 21 Evaluation of the discrimination power of such scores is possible by using the area under ROC curve,21 but for calibration this interpretation has been quite different. Some researchers have considered the accordance of these two tests to be the most important criterion for acceptance of any of these scores and have even suggested that lacking an acceptable calibration, in spite of good discrimination power, results in rejection of a scoring system.21
According to our results and in spite of the high correlation with APACHE II and SAPS II scores, due to the stronger calibration and discrimination power, APACHE II is considered to be a better tool for outcome evaluation in our respiratory ICU. In addition, studying the correlation between APACHE II/SAPS II scores and the actual death rate revealed that a stronger relation existed between APACHE II and actual death rate, although a light correlation also existed for SAPS II.
5. Conclusion
According to the results of this study, APACHE II score is considered acceptable for initial evaluation of patients admitted to a respiratory ICU.