ROC Optimisation of Safety Related Systems

Abstract

Many safety related and critical systems warn of potentially dangerous events; for example the Short Term Conflict Alert (STCA) system warns of airspace infractions between aircraft. Although installed with current technology such critical systems may become out of date due to changes in the circumstances in which they function, operational procedures and the regulatory environment. Current practice is to ‘tune’ by hand the many parameters governing the system in order to optimise the operating point in terms of the true positive and false positive rates, which are frequently associated with highly imbalanced costs. In this paper we cast the tuning of critical systems as a multiobjective optimisation problem. We show how a region of the optimal receiver operating characteristic (ROC) curve may be obtained, permitting the system operators to select the operating point. We apply this methodology to the STCA system, using a multi-objective (1 + 1)-evolution strategy, showing that we can improve upon the current hand-tuned operating point as well as providing the salient ROC curve describing the true-positive versus false-positive tradeoff.