Information-theoretic feature selection for classification

Alok A. Joshi, Scott M. James, Peter H. Meckl, Galen B. King, Kristofer Jennings

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Feature selection has always been an important aspect of statistical model identification and pattern classification. In this paper we introduce a novel information-theoretic index called the compensated quality factor (CQF) which selects the important features from a large amount of irrelevant data. The proposed index does an exhaustive combinatorial search of the input space and selects the feature that maximizes the information criterion conditioned on the decision rules defined by the compensated quality factor. The effectiveness of the proposed CQF-based algorithm was tested against the results of Mallows Cp criterion, Akaike information criterion (AIC), and Bayesian information criterion (BIC) using post liver operation survival data [1] (continuous variables) and NIST sonoluminescent light intensity data [2] (categorical variables). Due to computational time and memory constraints, the CQF-based feature selector is only recommended for an input space with dimension p < 20. The problem of higher dimensional input spaces (20 < p < 50) was solved by proposing an information-theoretic stepwise selection procedure. Though this procedure does not guarantee a globally optimal solution, the computational time-memory requirements are reduced drastically compared to the exhaustive combinatorial search. Using diesel engine data for fault detection (43 variables, 8-classes, 30000 records), the performance of the information-theoretic selection technique was tested by comparing the misclassification rates before and after the dimension reduction using various classifiers.

Original languageEnglish (US)
Title of host publicationProceedings of the American Control Conference
Pages2000-2005
Number of pages6
DOIs
StatePublished - 2007
Externally publishedYes
Event2007 American Control Conference, ACC - New York, NY, United States
Duration: Jul 9 2007Jul 13 2007

Other

Other2007 American Control Conference, ACC
CountryUnited States
CityNew York, NY
Period7/9/077/13/07

Fingerprint

Feature extraction
Data storage equipment
Fault detection
Liver
Pattern recognition
Diesel engines
Classifiers
Statistical Models

ASJC Scopus subject areas

  • Control and Systems Engineering

Cite this

Joshi, A. A., James, S. M., Meckl, P. H., King, G. B., & Jennings, K. (2007). Information-theoretic feature selection for classification. In Proceedings of the American Control Conference (pp. 2000-2005). [4282270] https://doi.org/10.1109/ACC.2007.4282270

Information-theoretic feature selection for classification. / Joshi, Alok A.; James, Scott M.; Meckl, Peter H.; King, Galen B.; Jennings, Kristofer.

Proceedings of the American Control Conference. 2007. p. 2000-2005 4282270.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Joshi, AA, James, SM, Meckl, PH, King, GB & Jennings, K 2007, Information-theoretic feature selection for classification. in Proceedings of the American Control Conference., 4282270, pp. 2000-2005, 2007 American Control Conference, ACC, New York, NY, United States, 7/9/07. https://doi.org/10.1109/ACC.2007.4282270
Joshi AA, James SM, Meckl PH, King GB, Jennings K. Information-theoretic feature selection for classification. In Proceedings of the American Control Conference. 2007. p. 2000-2005. 4282270 https://doi.org/10.1109/ACC.2007.4282270
Joshi, Alok A. ; James, Scott M. ; Meckl, Peter H. ; King, Galen B. ; Jennings, Kristofer. / Information-theoretic feature selection for classification. Proceedings of the American Control Conference. 2007. pp. 2000-2005
@inproceedings{69dc6ec16c6044d9b8f888b7613d1b9f,
title = "Information-theoretic feature selection for classification",
abstract = "Feature selection has always been an important aspect of statistical model identification and pattern classification. In this paper we introduce a novel information-theoretic index called the compensated quality factor (CQF) which selects the important features from a large amount of irrelevant data. The proposed index does an exhaustive combinatorial search of the input space and selects the feature that maximizes the information criterion conditioned on the decision rules defined by the compensated quality factor. The effectiveness of the proposed CQF-based algorithm was tested against the results of Mallows Cp criterion, Akaike information criterion (AIC), and Bayesian information criterion (BIC) using post liver operation survival data [1] (continuous variables) and NIST sonoluminescent light intensity data [2] (categorical variables). Due to computational time and memory constraints, the CQF-based feature selector is only recommended for an input space with dimension p < 20. The problem of higher dimensional input spaces (20 < p < 50) was solved by proposing an information-theoretic stepwise selection procedure. Though this procedure does not guarantee a globally optimal solution, the computational time-memory requirements are reduced drastically compared to the exhaustive combinatorial search. Using diesel engine data for fault detection (43 variables, 8-classes, 30000 records), the performance of the information-theoretic selection technique was tested by comparing the misclassification rates before and after the dimension reduction using various classifiers.",
author = "Joshi, {Alok A.} and James, {Scott M.} and Meckl, {Peter H.} and King, {Galen B.} and Kristofer Jennings",
year = "2007",
doi = "10.1109/ACC.2007.4282270",
language = "English (US)",
isbn = "1424409888",
pages = "2000--2005",
booktitle = "Proceedings of the American Control Conference",

}

TY - GEN

T1 - Information-theoretic feature selection for classification

AU - Joshi, Alok A.

AU - James, Scott M.

AU - Meckl, Peter H.

AU - King, Galen B.

AU - Jennings, Kristofer

PY - 2007

Y1 - 2007

N2 - Feature selection has always been an important aspect of statistical model identification and pattern classification. In this paper we introduce a novel information-theoretic index called the compensated quality factor (CQF) which selects the important features from a large amount of irrelevant data. The proposed index does an exhaustive combinatorial search of the input space and selects the feature that maximizes the information criterion conditioned on the decision rules defined by the compensated quality factor. The effectiveness of the proposed CQF-based algorithm was tested against the results of Mallows Cp criterion, Akaike information criterion (AIC), and Bayesian information criterion (BIC) using post liver operation survival data [1] (continuous variables) and NIST sonoluminescent light intensity data [2] (categorical variables). Due to computational time and memory constraints, the CQF-based feature selector is only recommended for an input space with dimension p < 20. The problem of higher dimensional input spaces (20 < p < 50) was solved by proposing an information-theoretic stepwise selection procedure. Though this procedure does not guarantee a globally optimal solution, the computational time-memory requirements are reduced drastically compared to the exhaustive combinatorial search. Using diesel engine data for fault detection (43 variables, 8-classes, 30000 records), the performance of the information-theoretic selection technique was tested by comparing the misclassification rates before and after the dimension reduction using various classifiers.

AB - Feature selection has always been an important aspect of statistical model identification and pattern classification. In this paper we introduce a novel information-theoretic index called the compensated quality factor (CQF) which selects the important features from a large amount of irrelevant data. The proposed index does an exhaustive combinatorial search of the input space and selects the feature that maximizes the information criterion conditioned on the decision rules defined by the compensated quality factor. The effectiveness of the proposed CQF-based algorithm was tested against the results of Mallows Cp criterion, Akaike information criterion (AIC), and Bayesian information criterion (BIC) using post liver operation survival data [1] (continuous variables) and NIST sonoluminescent light intensity data [2] (categorical variables). Due to computational time and memory constraints, the CQF-based feature selector is only recommended for an input space with dimension p < 20. The problem of higher dimensional input spaces (20 < p < 50) was solved by proposing an information-theoretic stepwise selection procedure. Though this procedure does not guarantee a globally optimal solution, the computational time-memory requirements are reduced drastically compared to the exhaustive combinatorial search. Using diesel engine data for fault detection (43 variables, 8-classes, 30000 records), the performance of the information-theoretic selection technique was tested by comparing the misclassification rates before and after the dimension reduction using various classifiers.

UR - http://www.scopus.com/inward/record.url?scp=46449100115&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=46449100115&partnerID=8YFLogxK

U2 - 10.1109/ACC.2007.4282270

DO - 10.1109/ACC.2007.4282270

M3 - Conference contribution

SN - 1424409888

SN - 9781424409884

SP - 2000

EP - 2005

BT - Proceedings of the American Control Conference

ER -