Expected Performance Curves

Description

In several research domains concerned with classification tasks, including person authentication and text categorization, ROC curves are often used to assess the quality of a particular model or to compare two or more models. Researchers also often publish some statistics coming from the ROC curve, such as so-called break-even point or equal error rate. The purpose of this paper is to argue that these measures can be misleading and should be avoided. Furthermore, we propose a replacement for them, called Expected Performance Curves (EPC) which enable the comparison of several systems according to a criterion, decided by the application, which is used to set thresholds according to a separate validation set.

Related technical report

S. Bengio, M. Keller, and J. Mariéthoz. The expected performance curve. Technical Report IDIAP-RR 03-85, IDIAP, 2003. Submitted for publication.
[ .pdf | .djvu | .ps.gz ]

Software

This software is under the same license as Torch, hence under a BSD license.

Dowloads
    Archive     Description Documentation
Torch3 EPC package Torch3 EPC package: can compute EPC curves readme epc
EPC examples EPC biometric verification example using gnuplot readme examples
qtplot    Plotting program written in Qt    readme qtplot

Example of EPC curve

Figure Examples
gnuplot qtplot
Figure generated by gnuplot Figure generated by qtplot (zoom)

Authors

Samy Bengio, Mikaela Keller, Johnny Mariéthoz