Facilitating Clinical Outcomes Assessment through the Automated Identification of Quality Measures for Prostate Cancer Surgery

Leonard W. D'Avolio, Mark S. Litwin, Selwyn O. Rogers, Alex A T Bui

Research output: Contribution to journalArticle

12 Citations (Scopus)

Abstract

Objectives: The College of American Pathologists (CAP) Category 1 quality measures, tumor stage, Gleason score, and surgical margin status, are used by physicians and cancer registrars to categorize patients into groups for clinical trials and treatment planning. This study was conducted to evaluate the effectiveness of an application designed to automatically extract these quality measures from the postoperative pathology reports of patients having undergone prostatectomies for treatment of prostate cancer. Design: An application was developed with the Clinical Outcomes Assessment Toolkit that uses an information pipeline of regular expressions and support vector machines to extract CAP Category 1 quality measures. System performance was evaluated against a gold standard of 676 pathology reports from the University of California at Los Angeles Medical Center and Brigham and Women's Hospital. To evaluate the feasibility of clinical implementation, all pathology reports were gathered using administrative codes with no manual preprocessing of the data performed. Measurements: The sensitivity, specificity, and overall accuracy of system performance were measured for all three quality measures. Performance at both hospitals was compared, and a detailed failure analysis was conducted to identify errors caused by poor data quality versus system shortcomings. Results: Accuracies for Gleason score were 99.7%, tumor stage 99.1%, and margin status 97.2%, for an overall accuracy of 98.67%. System performance on data from both hospitals was comparable. Poor clinical data quality led to a decrease in overall accuracy of only 0.3% but accounted for 25.9% of the total errors. Conclusion: Despite differences in document format and pathologists' reporting styles, strong system performance indicates the potential of using a combination of regular expressions and support vector machines to automatically extract CAP Category 1 quality measures from postoperative prostate cancer pathology reports.

Original languageEnglish (US)
Pages (from-to)341-348
Number of pages8
JournalJournal of the American Medical Informatics Association
Volume15
Issue number3
DOIs
StatePublished - May 2008
Externally publishedYes

Fingerprint

Prostatic Neoplasms
Outcome Assessment (Health Care)
Pathology
Neoplasm Grading
Neoplasms
Los Angeles
Prostatectomy
Clinical Trials
Physicians
Sensitivity and Specificity
Pathologists
Therapeutics
Data Accuracy
Support Vector Machine

ASJC Scopus subject areas

  • Medicine(all)

Cite this

Facilitating Clinical Outcomes Assessment through the Automated Identification of Quality Measures for Prostate Cancer Surgery. / D'Avolio, Leonard W.; Litwin, Mark S.; Rogers, Selwyn O.; Bui, Alex A T.

In: Journal of the American Medical Informatics Association, Vol. 15, No. 3, 05.2008, p. 341-348.

Research output: Contribution to journalArticle

D'Avolio, Leonard W. ; Litwin, Mark S. ; Rogers, Selwyn O. ; Bui, Alex A T. / Facilitating Clinical Outcomes Assessment through the Automated Identification of Quality Measures for Prostate Cancer Surgery. In: Journal of the American Medical Informatics Association. 2008 ; Vol. 15, No. 3. pp. 341-348.
@article{0fbf1e91f9674204bf2048d96954ff4b,
title = "Facilitating Clinical Outcomes Assessment through the Automated Identification of Quality Measures for Prostate Cancer Surgery",
abstract = "Objectives: The College of American Pathologists (CAP) Category 1 quality measures, tumor stage, Gleason score, and surgical margin status, are used by physicians and cancer registrars to categorize patients into groups for clinical trials and treatment planning. This study was conducted to evaluate the effectiveness of an application designed to automatically extract these quality measures from the postoperative pathology reports of patients having undergone prostatectomies for treatment of prostate cancer. Design: An application was developed with the Clinical Outcomes Assessment Toolkit that uses an information pipeline of regular expressions and support vector machines to extract CAP Category 1 quality measures. System performance was evaluated against a gold standard of 676 pathology reports from the University of California at Los Angeles Medical Center and Brigham and Women's Hospital. To evaluate the feasibility of clinical implementation, all pathology reports were gathered using administrative codes with no manual preprocessing of the data performed. Measurements: The sensitivity, specificity, and overall accuracy of system performance were measured for all three quality measures. Performance at both hospitals was compared, and a detailed failure analysis was conducted to identify errors caused by poor data quality versus system shortcomings. Results: Accuracies for Gleason score were 99.7{\%}, tumor stage 99.1{\%}, and margin status 97.2{\%}, for an overall accuracy of 98.67{\%}. System performance on data from both hospitals was comparable. Poor clinical data quality led to a decrease in overall accuracy of only 0.3{\%} but accounted for 25.9{\%} of the total errors. Conclusion: Despite differences in document format and pathologists' reporting styles, strong system performance indicates the potential of using a combination of regular expressions and support vector machines to automatically extract CAP Category 1 quality measures from postoperative prostate cancer pathology reports.",
author = "D'Avolio, {Leonard W.} and Litwin, {Mark S.} and Rogers, {Selwyn O.} and Bui, {Alex A T}",
year = "2008",
month = "5",
doi = "10.1197/jamia.M2649",
language = "English (US)",
volume = "15",
pages = "341--348",
journal = "Journal of the American Medical Informatics Association : JAMIA",
issn = "1067-5027",
publisher = "Oxford University Press",
number = "3",

}

TY - JOUR

T1 - Facilitating Clinical Outcomes Assessment through the Automated Identification of Quality Measures for Prostate Cancer Surgery

AU - D'Avolio, Leonard W.

AU - Litwin, Mark S.

AU - Rogers, Selwyn O.

AU - Bui, Alex A T

PY - 2008/5

Y1 - 2008/5

N2 - Objectives: The College of American Pathologists (CAP) Category 1 quality measures, tumor stage, Gleason score, and surgical margin status, are used by physicians and cancer registrars to categorize patients into groups for clinical trials and treatment planning. This study was conducted to evaluate the effectiveness of an application designed to automatically extract these quality measures from the postoperative pathology reports of patients having undergone prostatectomies for treatment of prostate cancer. Design: An application was developed with the Clinical Outcomes Assessment Toolkit that uses an information pipeline of regular expressions and support vector machines to extract CAP Category 1 quality measures. System performance was evaluated against a gold standard of 676 pathology reports from the University of California at Los Angeles Medical Center and Brigham and Women's Hospital. To evaluate the feasibility of clinical implementation, all pathology reports were gathered using administrative codes with no manual preprocessing of the data performed. Measurements: The sensitivity, specificity, and overall accuracy of system performance were measured for all three quality measures. Performance at both hospitals was compared, and a detailed failure analysis was conducted to identify errors caused by poor data quality versus system shortcomings. Results: Accuracies for Gleason score were 99.7%, tumor stage 99.1%, and margin status 97.2%, for an overall accuracy of 98.67%. System performance on data from both hospitals was comparable. Poor clinical data quality led to a decrease in overall accuracy of only 0.3% but accounted for 25.9% of the total errors. Conclusion: Despite differences in document format and pathologists' reporting styles, strong system performance indicates the potential of using a combination of regular expressions and support vector machines to automatically extract CAP Category 1 quality measures from postoperative prostate cancer pathology reports.

AB - Objectives: The College of American Pathologists (CAP) Category 1 quality measures, tumor stage, Gleason score, and surgical margin status, are used by physicians and cancer registrars to categorize patients into groups for clinical trials and treatment planning. This study was conducted to evaluate the effectiveness of an application designed to automatically extract these quality measures from the postoperative pathology reports of patients having undergone prostatectomies for treatment of prostate cancer. Design: An application was developed with the Clinical Outcomes Assessment Toolkit that uses an information pipeline of regular expressions and support vector machines to extract CAP Category 1 quality measures. System performance was evaluated against a gold standard of 676 pathology reports from the University of California at Los Angeles Medical Center and Brigham and Women's Hospital. To evaluate the feasibility of clinical implementation, all pathology reports were gathered using administrative codes with no manual preprocessing of the data performed. Measurements: The sensitivity, specificity, and overall accuracy of system performance were measured for all three quality measures. Performance at both hospitals was compared, and a detailed failure analysis was conducted to identify errors caused by poor data quality versus system shortcomings. Results: Accuracies for Gleason score were 99.7%, tumor stage 99.1%, and margin status 97.2%, for an overall accuracy of 98.67%. System performance on data from both hospitals was comparable. Poor clinical data quality led to a decrease in overall accuracy of only 0.3% but accounted for 25.9% of the total errors. Conclusion: Despite differences in document format and pathologists' reporting styles, strong system performance indicates the potential of using a combination of regular expressions and support vector machines to automatically extract CAP Category 1 quality measures from postoperative prostate cancer pathology reports.

UR - http://www.scopus.com/inward/record.url?scp=43049090471&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=43049090471&partnerID=8YFLogxK

U2 - 10.1197/jamia.M2649

DO - 10.1197/jamia.M2649

M3 - Article

C2 - 18308980

AN - SCOPUS:43049090471

VL - 15

SP - 341

EP - 348

JO - Journal of the American Medical Informatics Association : JAMIA

JF - Journal of the American Medical Informatics Association : JAMIA

SN - 1067-5027

IS - 3

ER -