Conventional risk prediction models fail to accurately predict mortality risk among patients with coronavirus disease 2019 in intensive care units: a difficult time to assess clinical severity and quality of care
Journal of Intensive Care volume 9, Article number: 42 (2021)
Since the start of the coronavirus disease 2019 (COVID-19) pandemic, it has remained unknown whether conventional risk prediction tools used in intensive care units are applicable to patients with COVID-19. Therefore, we assessed the performance of established risk prediction models using the Japanese Intensive Care database. Discrimination and calibration of the models were poor. Revised risk prediction models are needed to assess the clinical severity of COVID-19 patients and monitor healthcare quality in ICUs overwhelmed by patients with COVID-19.
Since the start of the coronavirus disease 2019 (COVID-19) pandemic, intensive care units (ICUs) worldwide have struggled to treat affected patients who require a completely different approach to treatment than other patients . Although many severe cases are admitted to ICUs, it is unknown whether the conventional risk scoring systems that were developed for ICU patients can be applied to patients with COVID-19. With unknown predictive performance, healthcare professionals have faced difficulties in assessing the clinical severity of patients with COVID-19 and monitoring the quality of care in ICUs. New risk prediction models for COVID-19 patients have been developed , but most of these were not developed specifically for ICU patients, and it is unknown whether they perform as well in clinical practice as they did in the model development studies. It is also likely that overwhelmed ICUs lack the time to derive and validate novel risk scores. In such circumstances, ICUs must use conventional scoring systems, such as the Acute Physiology and Chronic Health Evaluation (APACHE) and Simplified Acute Physiology Score (SAPS). Several recent studies have used APACHE and SAPS to provide information on the clinical severity of COVID-19 [3,4,5]. However, very few reports have examined their validity of applying them to patients with COVID-19. One letter from the UK reported that APACHE II underestimated the risk of death, concluding that the risk scoring systems that were widely used before the pandemic were inappropriate for evaluating the clinical severity of COVID-19 . In Japan, a research group recently developed the Japan Risk of Death (JROD), a prediction model that recalibrated the APACHE III-j model . However, this model may show limited validity in patients with COVID-19 because it was developed using the data collected before the pandemic and it was designed for general use in ICUs. Therefore, we investigated whether conventional risk prediction models, such as APACHE II, SAPS II, APACHE III-j, and JROD, can be applied to patients with COVID-19 and determined their predictive performance.
We obtained data for confirmed cases of COVID-19 admitted between January 2020 and February 2021 from the Japanese Intensive Care Patient Database (JIPAD) . We used JROD to predict mortality in the same way as in the previous study , but with a development period of January 2019 to December 2019. This was then applied to predict mortality in the study cohort and defined as JROD2019 predicted mortality. The predictive performances of APACHE II, SAPS II, APACHE III-j, and JROD2019 were assessed using the area under the receiver operating characteristic curves, Brier scores, Hosmer–Lemeshow tests, calibration plots, and standardized mortality ratios.
A total of 444 patients admitted to 40 ICUs in Japan were extracted from the JIPAD for analysis. The clinical characteristics of patients are shown in Table 1. The model performance statistics are presented in Table 2 and Fig. 1. Death at hospital discharge was recorded in 69 patients (15.5%), which was less than half the mortality reported by Stephens et al., although the APACHE II scores were comparable . Using JIPAD data, the APACHE II, SAPS II, and APACHE III-j models overestimated the risk of death, whereas JROD2019 underestimated the risk. The discrimination and calibration of APACHE III-j and JROD were poor compared with those reported in the JROD development study . Although the results are dissimilar to a previous report  in terms of the direction of estimated risk (i.e., overestimation/underestimation), we make the same conclusion that the risk models used before the pandemic are not suitable for patients with COVID-19. Of note, even JROD2019, a model that was developed to improve the predictive ability of APACHE III-j, displayed suboptimal predictive performance. Owing to the poor predictive performance, it is difficult to incorporate the predicted mortality calculated using these risk models in quality assessment tools, such as funnel plots and exponentially weighted moving average charts, with high reliability. Consequently, it will be difficult to implement quality assessment and improvement in ICUs, particularly those where patients with COVID-19 occupy a high proportion of ICU beds. Calibration can be improved with simple update methods, like that done in the JROD study, but discrimination can only be improved by updating the coefficients of each predictor and/or adding other relevant predictors . Thus, a revised risk prediction model designed specifically for COVID-19 patients together with logistical support for its implementation in ICUs are urgently needed.
Availability of data and materials
The authors’ agreement with the JIPAD project does not allow us to publish the data used for this manuscript or to share it with others.
Acute Physiology and Chronic Health Evaluation
Coronavirus disease 2019
Intensive care unit
Japanese Intensive Care Patient Database
Japan Risk of Death
Simplified Acute Physiology Score
Ma X, Vervoort D. Critical care capacity during the COVID-19 pandemic: global availability of intensive care beds. J Crit Care. 2020;58:96–7. https://doi.org/10.1016/j.jcrc.2020.04.012.
Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020;369:m1328. https://doi.org/10.1136/bmj.m1328.
Xie J, Wu W, Li S, Hu Y, Hu M, Li J, et al. Clinical characteristics and outcomes of critically ill patients with novel coronavirus infectious disease (COVID-19) in China: a retrospective multicenter study. Intensive Care Med. 2020;46(10):1863–72. https://doi.org/10.1007/s00134-020-06211-2.
Ferrando C, Mellado-Artigas R, Gea A, Arruti E, Aldecoa C, Bordell A, et al. Patient characteristics, clinical course and factors associated to ICU mortality in critically ill patients infected with SARS-CoV-2 in Spain: a prospective, cohort, multicentre study. Rev Esp Anestesiol Reanim. 2020;67(8):425–37. https://doi.org/10.1016/j.redar.2020.07.003.
COVID-ICU Group on behalf of the REVA Network and the COVID-ICU Investigators. Clinical characteristics and day-90 outcomes of 4244 critically ill adults with COVID-19: a prospective cohort study. Intensive Care Med. 2021;47:60–73. https://doi.org/10.1007/s00134-020-06294-x.
Stephens JR, Stümpfle R, Patel P, Brett S, Broomhead R, Baharlo B, et al. Analysis of critical care severity of illness scoring systems in patients with coronavirus disease 2019: a retrospective analysis of three U.K. ICUs. Crit Care Med. 2021;49(1):e105–7. https://doi.org/10.1097/CCM.0000000000004674.
Endo H, Uchino S, Hashimoto S, Aoki Y, Hashiba E, Hatakeyama J, et al. Development and validation of the predictive risk of death model for adult patients admitted to intensive care units in Japan: an approach to improve the accuracy of healthcare quality measures. J Intensive Care. 2021;9(1):18. https://doi.org/10.1186/s40560-021-00533-z.
Irie H, Okamoto H, Uchino S, Endo H, Uchida M, Kawasaki T, et al. The Japanese Intensive care PAtient Database (JIPAD): a national intensive care unit registry in Japan. J Crit Care. 2019;55:86–94.https://doi.org/10.1016/j.jcrc.2019.09.004.
Steyerberg EW. Clinical prediction models. 2nd ed. Springer Nature: Cham; 2019. https://doi.org/10.1007/978-3-030-16399-0.
We would like to thank all of the hospitals participating in the JIPAD for their contribution.
This paper was written as a part of the JIPAD project, which was funded by the Japanese Society of Intensive Care Medicine.
Ethics approval and consent to participate
This study was approved by the Research Ethics Committee of the University of Tokyo (Approval number: 2020242NI-(1)). Consent to participate was waived because of the anonymous nature of the data.
Consent for publication
HE, NI, and HM are affiliated with the Department of Healthcare Quality Assessment at the University of Tokyo. The department is a social collaboration department supported by grants from the National Clinical Database, Johnson & Johnson K.K., and Nipro Corporation. The other authors do not have any competing interests to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Endo, H., Ohbe, H., Kumasawa, J. et al. Conventional risk prediction models fail to accurately predict mortality risk among patients with coronavirus disease 2019 in intensive care units: a difficult time to assess clinical severity and quality of care. j intensive care 9, 42 (2021). https://doi.org/10.1186/s40560-021-00557-5
- Coronavirus disease 2019
- Risk of death
- Intensive care unit
- Risk prediction model
- Quality improvement