Effect of Human-AI Interaction on Detection of Malignant Lung Nodules on Chest Radiographs

Abstract

Background: The factors affecting radiologists diagnostic determinations in artificial intelligence (AI)–assisted image reading remain underexplored. Purpose: To assess how AI diagnostic performance and reader characteristics influence detection of malignant lung nodules during AI-assisted reading of chest radiographs. Materials and Methods: This retrospective study consisted of two reading sessions from April 2021 to June 2021. Based on the first session without AI assistance, 30 readers were assigned into two groups with equivalent areas under the free-response receiver operating characteristic curve (AUFROCs). In the second session, each group reinterpreted radiographs assisted by either a high or low accuracy AI model (blinded to the fact that two different AI models were used). Reader performance for detecting lung cancer and reader susceptibility (changing the original reading following the AI suggestion) were compared. A generalized linear mixed model was used to identify the factors influencing AI-assisted detection performance, including readers attitudes and experiences of AI and Grit score. Results: Of the 120 chest radiographs assessed, 60 were obtained in patients with lung cancer (mean age, 67 years ± 12 [SD]; 32 male; 63 cancers) and 60 in controls (mean age, 67 years ± 12; 36 male). Readers included 20 thoracic radiologists (5–18 years of experience) and 10 radiology residents (2–3 years of experience). Use of the high accuracy AI model improved readers detection performance to a greater extent than use of the low accuracy AI model (area under the receiver operating characteristic curve, 0.77 to 0.82 vs 0.75 to 0.75; AUFROC, 0.71 to 0.79 vs 0.7 to 0.72). Readers who used the high accuracy AI showed a higher susceptibility (67%, 224 of 334 cases) to changing their diagnosis based on the AI suggestions than those using the low accuracy AI (59%, 229 of 386 cases). Accurate readings at the first session, correct AI suggestions, high accuracy Al, and diagnostic difficulty were associated with accurate AI-assisted readings, but readers characteristics were not. Conclusion: An AI model with high diagnostic accuracy led to improved performance of radiologists in detecting lung cancer on chest radiographs and increased radiologists susceptibility to AI suggestions.