Inter-rater reliability and prospective validation of a clinical prediction rule for SARS-CoV-2 infection

Abstract

Objectives Accurate estimation of the risk of SARS-CoV-2 infection based on bedside data alone has importance to Emergency Department (ED) operations and throughput. The 13 item CORC (COVID [or coronavirus] Rule-out Criteria) rule had good overall diagnostic accuracy in retrospective derivation and validation. The objective of this study was to prospectively test the inter-rater-reliability and diagnostic accuracy of the CORC score and rule (score ≤0 negative, >0 positive), and compare the CORC rule performance with physician gestalt. Methods This non-interventional study was conducted at an urban academic ED from February-March 2021. Two practitioners were approached by research coordinators and asked to independently complete a form capturing the CORC criteria for their shared patient, and their gestalt binary prediction of the SARS-CoV-2 test result and confidence (0-100%). The criterion standard for SARS-CoV-2 was from rt-PCR performed on a nasopharyngeal swab. The primary analysis was from weighted Cohen’s K and likelihood ratios (LR). Results For 928 patients, agreement between observers was good for the total CORC score, K = 0.613 (0.579-0.646) and for the CORC rule K = 0.644 (0.591-0.697). The agreement for clinician gestalt binary determination of SARs-CoV-2 status was K = 0.534, (95% CI 0.437-0.632) with median confidence of 76% (1st-3rd quartile: 66-88.5). For 425 patients who had the criterion standard, a negative CORC rule (both observers scored CORC <0), the sensitivity was 88%, specificity 51%, with a LR(-) = 0.24 (95% CI 0.10-0.50). Among patients with a mean CORC score >4, the prevalence of a positive SARS-CoV-2 test was 58% (28-85%) and LR(+) = 13.1 (4.5-37.2). Clinician gestalt demonstrated a sensitivity of 51%, specificity of 86% with a LR(-) = 0.57 (0.39-0.74). Conclusion In this prospective study, the CORC score and rule demonstrated good inter-rater reliability and reproducible diagnostic accuracy for estimating the pretest probability of SARs-CoV-2 infection.