Effect of CAD on Radiologists' Detection of Lung Nodules on Thoracic CT Scans

Analysis of an Observer Performance Study by Nodule Size

Berkman Sahiner, Heang Ping Chan, Lubomir M. Hadjiiski, Philip N. Cascade, Ella A. Kazerooni, Aamer R. Chughtai, Chad Poopat, Thomas Song, Luba Frank, Jadranka Stojanovska, Anil Attili

Research output: Contribution to journalArticle

65 Citations (Scopus)

Abstract

Rationale and Objectives: To retrospectively investigate the effect of a computer-aided detection (CAD) system on radiologists' performance for detecting small pulmonary nodules in computed tomography (CT) examinations, with a panel of expert radiologists serving as the reference standard. Materials and Methods: Institutional review board approval was obtained. Our dataset contained 52 CT examinations collected by the Lung Image Database Consortium, and 33 from our institution. All CTs were read by multiple expert thoracic radiologists to identify the reference standard for detection. Six other thoracic radiologists read the CT examinations first without and then with CAD. Performance was evaluated using free-response receiver operating characteristics (FROC) and the jackknife FROC analysis methods (JAFROC) for nodules above different diameter thresholds. Results: A total of 241 nodules, ranging in size from 3.0 to 18.6 mm (mean, 5.3 mm) were identified as the reference standard. At diameter thresholds of 3, 4, 5, and 6 mm, the CAD system had a sensitivity of 54%, 64%, 68%, and 76%, respectively, with an average of 5.6 false positives (FPs) per scan. Without CAD, the average figures of merit (FOMs) for the six radiologists, obtained from JAFROC analysis, were 0.661, 0.729, 0.793, and 0.838 for the same nodule diameter thresholds, respectively. With CAD, the corresponding average FOMs improved to 0.705, 0.763, 0.810, and 0.862, respectively. The improvement achieved statistical significance for nodules at the 3 and 4 mm thresholds (P = .002 and .020, respectively), and did not achieve significance at 5 and 6 mm (P = .18 and .13, respectively). At a nodule diameter threshold of 3 mm, the radiologists' average sensitivity and FP rate were 0.56 and 0.67, respectively, without CAD, and 0.67 and 0.78 with CAD. Conclusion: CAD improves thoracic radiologists' performance for detecting pulmonary nodules smaller than 5 mm on CT examinations, which are often overlooked by visual inspection alone.

Original languageEnglish (US)
Pages (from-to)1518-1530
Number of pages13
JournalAcademic Radiology
Volume16
Issue number12
DOIs
StatePublished - Dec 2009
Externally publishedYes

Fingerprint

Thorax
Tomography
Lung
ROC Curve
Radiologists
Research Ethics Committees
Databases

Keywords

  • computer-aided detection
  • CT
  • Lung Nodule

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Cite this

Sahiner, B., Chan, H. P., Hadjiiski, L. M., Cascade, P. N., Kazerooni, E. A., Chughtai, A. R., ... Attili, A. (2009). Effect of CAD on Radiologists' Detection of Lung Nodules on Thoracic CT Scans: Analysis of an Observer Performance Study by Nodule Size. Academic Radiology, 16(12), 1518-1530. https://doi.org/10.1016/j.acra.2009.08.006

Effect of CAD on Radiologists' Detection of Lung Nodules on Thoracic CT Scans : Analysis of an Observer Performance Study by Nodule Size. / Sahiner, Berkman; Chan, Heang Ping; Hadjiiski, Lubomir M.; Cascade, Philip N.; Kazerooni, Ella A.; Chughtai, Aamer R.; Poopat, Chad; Song, Thomas; Frank, Luba; Stojanovska, Jadranka; Attili, Anil.

In: Academic Radiology, Vol. 16, No. 12, 12.2009, p. 1518-1530.

Research output: Contribution to journalArticle

Sahiner, B, Chan, HP, Hadjiiski, LM, Cascade, PN, Kazerooni, EA, Chughtai, AR, Poopat, C, Song, T, Frank, L, Stojanovska, J & Attili, A 2009, 'Effect of CAD on Radiologists' Detection of Lung Nodules on Thoracic CT Scans: Analysis of an Observer Performance Study by Nodule Size', Academic Radiology, vol. 16, no. 12, pp. 1518-1530. https://doi.org/10.1016/j.acra.2009.08.006
Sahiner, Berkman ; Chan, Heang Ping ; Hadjiiski, Lubomir M. ; Cascade, Philip N. ; Kazerooni, Ella A. ; Chughtai, Aamer R. ; Poopat, Chad ; Song, Thomas ; Frank, Luba ; Stojanovska, Jadranka ; Attili, Anil. / Effect of CAD on Radiologists' Detection of Lung Nodules on Thoracic CT Scans : Analysis of an Observer Performance Study by Nodule Size. In: Academic Radiology. 2009 ; Vol. 16, No. 12. pp. 1518-1530.
@article{4aa286a657184bbc8277e7bbc21a4ca9,
title = "Effect of CAD on Radiologists' Detection of Lung Nodules on Thoracic CT Scans: Analysis of an Observer Performance Study by Nodule Size",
abstract = "Rationale and Objectives: To retrospectively investigate the effect of a computer-aided detection (CAD) system on radiologists' performance for detecting small pulmonary nodules in computed tomography (CT) examinations, with a panel of expert radiologists serving as the reference standard. Materials and Methods: Institutional review board approval was obtained. Our dataset contained 52 CT examinations collected by the Lung Image Database Consortium, and 33 from our institution. All CTs were read by multiple expert thoracic radiologists to identify the reference standard for detection. Six other thoracic radiologists read the CT examinations first without and then with CAD. Performance was evaluated using free-response receiver operating characteristics (FROC) and the jackknife FROC analysis methods (JAFROC) for nodules above different diameter thresholds. Results: A total of 241 nodules, ranging in size from 3.0 to 18.6 mm (mean, 5.3 mm) were identified as the reference standard. At diameter thresholds of 3, 4, 5, and 6 mm, the CAD system had a sensitivity of 54{\%}, 64{\%}, 68{\%}, and 76{\%}, respectively, with an average of 5.6 false positives (FPs) per scan. Without CAD, the average figures of merit (FOMs) for the six radiologists, obtained from JAFROC analysis, were 0.661, 0.729, 0.793, and 0.838 for the same nodule diameter thresholds, respectively. With CAD, the corresponding average FOMs improved to 0.705, 0.763, 0.810, and 0.862, respectively. The improvement achieved statistical significance for nodules at the 3 and 4 mm thresholds (P = .002 and .020, respectively), and did not achieve significance at 5 and 6 mm (P = .18 and .13, respectively). At a nodule diameter threshold of 3 mm, the radiologists' average sensitivity and FP rate were 0.56 and 0.67, respectively, without CAD, and 0.67 and 0.78 with CAD. Conclusion: CAD improves thoracic radiologists' performance for detecting pulmonary nodules smaller than 5 mm on CT examinations, which are often overlooked by visual inspection alone.",
keywords = "computer-aided detection, CT, Lung Nodule",
author = "Berkman Sahiner and Chan, {Heang Ping} and Hadjiiski, {Lubomir M.} and Cascade, {Philip N.} and Kazerooni, {Ella A.} and Chughtai, {Aamer R.} and Chad Poopat and Thomas Song and Luba Frank and Jadranka Stojanovska and Anil Attili",
year = "2009",
month = "12",
doi = "10.1016/j.acra.2009.08.006",
language = "English (US)",
volume = "16",
pages = "1518--1530",
journal = "Academic Radiology",
issn = "1076-6332",
publisher = "Elsevier USA",
number = "12",

}

TY - JOUR

T1 - Effect of CAD on Radiologists' Detection of Lung Nodules on Thoracic CT Scans

T2 - Analysis of an Observer Performance Study by Nodule Size

AU - Sahiner, Berkman

AU - Chan, Heang Ping

AU - Hadjiiski, Lubomir M.

AU - Cascade, Philip N.

AU - Kazerooni, Ella A.

AU - Chughtai, Aamer R.

AU - Poopat, Chad

AU - Song, Thomas

AU - Frank, Luba

AU - Stojanovska, Jadranka

AU - Attili, Anil

PY - 2009/12

Y1 - 2009/12

N2 - Rationale and Objectives: To retrospectively investigate the effect of a computer-aided detection (CAD) system on radiologists' performance for detecting small pulmonary nodules in computed tomography (CT) examinations, with a panel of expert radiologists serving as the reference standard. Materials and Methods: Institutional review board approval was obtained. Our dataset contained 52 CT examinations collected by the Lung Image Database Consortium, and 33 from our institution. All CTs were read by multiple expert thoracic radiologists to identify the reference standard for detection. Six other thoracic radiologists read the CT examinations first without and then with CAD. Performance was evaluated using free-response receiver operating characteristics (FROC) and the jackknife FROC analysis methods (JAFROC) for nodules above different diameter thresholds. Results: A total of 241 nodules, ranging in size from 3.0 to 18.6 mm (mean, 5.3 mm) were identified as the reference standard. At diameter thresholds of 3, 4, 5, and 6 mm, the CAD system had a sensitivity of 54%, 64%, 68%, and 76%, respectively, with an average of 5.6 false positives (FPs) per scan. Without CAD, the average figures of merit (FOMs) for the six radiologists, obtained from JAFROC analysis, were 0.661, 0.729, 0.793, and 0.838 for the same nodule diameter thresholds, respectively. With CAD, the corresponding average FOMs improved to 0.705, 0.763, 0.810, and 0.862, respectively. The improvement achieved statistical significance for nodules at the 3 and 4 mm thresholds (P = .002 and .020, respectively), and did not achieve significance at 5 and 6 mm (P = .18 and .13, respectively). At a nodule diameter threshold of 3 mm, the radiologists' average sensitivity and FP rate were 0.56 and 0.67, respectively, without CAD, and 0.67 and 0.78 with CAD. Conclusion: CAD improves thoracic radiologists' performance for detecting pulmonary nodules smaller than 5 mm on CT examinations, which are often overlooked by visual inspection alone.

AB - Rationale and Objectives: To retrospectively investigate the effect of a computer-aided detection (CAD) system on radiologists' performance for detecting small pulmonary nodules in computed tomography (CT) examinations, with a panel of expert radiologists serving as the reference standard. Materials and Methods: Institutional review board approval was obtained. Our dataset contained 52 CT examinations collected by the Lung Image Database Consortium, and 33 from our institution. All CTs were read by multiple expert thoracic radiologists to identify the reference standard for detection. Six other thoracic radiologists read the CT examinations first without and then with CAD. Performance was evaluated using free-response receiver operating characteristics (FROC) and the jackknife FROC analysis methods (JAFROC) for nodules above different diameter thresholds. Results: A total of 241 nodules, ranging in size from 3.0 to 18.6 mm (mean, 5.3 mm) were identified as the reference standard. At diameter thresholds of 3, 4, 5, and 6 mm, the CAD system had a sensitivity of 54%, 64%, 68%, and 76%, respectively, with an average of 5.6 false positives (FPs) per scan. Without CAD, the average figures of merit (FOMs) for the six radiologists, obtained from JAFROC analysis, were 0.661, 0.729, 0.793, and 0.838 for the same nodule diameter thresholds, respectively. With CAD, the corresponding average FOMs improved to 0.705, 0.763, 0.810, and 0.862, respectively. The improvement achieved statistical significance for nodules at the 3 and 4 mm thresholds (P = .002 and .020, respectively), and did not achieve significance at 5 and 6 mm (P = .18 and .13, respectively). At a nodule diameter threshold of 3 mm, the radiologists' average sensitivity and FP rate were 0.56 and 0.67, respectively, without CAD, and 0.67 and 0.78 with CAD. Conclusion: CAD improves thoracic radiologists' performance for detecting pulmonary nodules smaller than 5 mm on CT examinations, which are often overlooked by visual inspection alone.

KW - computer-aided detection

KW - CT

KW - Lung Nodule

UR - http://www.scopus.com/inward/record.url?scp=70350462590&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70350462590&partnerID=8YFLogxK

U2 - 10.1016/j.acra.2009.08.006

DO - 10.1016/j.acra.2009.08.006

M3 - Article

VL - 16

SP - 1518

EP - 1530

JO - Academic Radiology

JF - Academic Radiology

SN - 1076-6332

IS - 12

ER -