Confidence in Classification: A Bayesian Approach
Krzanowski, Wojtek J.
Bailey, Trevor C.
Fieldsend, Jonathan E.
Everson, Richard M.
Journal of Classification
Bayesian classification is currently of considerable interest. It provides a strategy for eliminating the uncertainty associated with a particular choice of classifiermodel parameters, and is the optimal decision-theoretic choice under certain circumstances when there is no single “true” classifier for a given data set. Modern computing capabilities can easily support the Markov chain Monte Carlo sampling that is necessary to carry out the calculations involved, but the information available in these samples is not at present being fully utilised. We show how it can be allied to known results concerning the “reject option” in order to produce an assessment of the confidence that can be ascribed to particular classifications, and how these confidence measures can be used to compare the performances of classifiers. Incorporating these confidence measures can alter the apparent ranking of classifiers as given by straightforward success or error rates. Several possible methods for obtaining confidence assessments are described, and compared on a range of data sets using the Bayesian probabilistic nearest-neighbour classifier.
Copyright © 2006 Springer Verlag. The final publication is available at link.springer.com
Notes: This paper was the first to present an in-depth study of methods for ascribing confidence levels to classification results derived using Bayesian averaging methods, and introduces several new techniques for this in a Bayesian context. These techniques allow measures of uncertainty in classification to be obtained by considering the disagreement in classification between models selected using Bayesian averaging methods. Such uncertainty measures are particularly important for safety-critical classification situations, such as those arising in automated medical classification and diagnosis.
Vol. 23 (2), pp. 199 - 220