Expected Performance Curves

Description

In several research domains concerned with classification tasks, including person authentication and text categorization, ROC curves are often used to assess the quality of a particular model or to compare two or more models. Researchers also often publish some statistics coming from the ROC curve, such as so-called break-even point or equal error rate. The purpose of this paper is to argue that these measures can be misleading and should be avoided. Furthermore, we propose a replacement for them, called Expected Performance Curves (EPC) which enable the comparison of several systems according to a criterion, decided by the application, which is used to set thresholds according to a separate validation set.

Related technical report

S. Bengio, M. Keller, and J. Mariéthoz. The expected performance curve. Technical Report IDIAP-RR 03-85, IDIAP, 2003. Submitted for publication.
[ .pdf | .djvu | .ps.gz ]

Software

This software is under the same license as Torch, hence under a BSD license.

Dowloads
Archive	Description	Documentation
Torch3 EPC package	Torch3 EPC package: can compute EPC curves	readme epc
EPC examples	EPC biometric verification example using gnuplot	readme examples
qtplot	Plotting program written in Qt	readme qtplot

Example of EPC curve

Figure Examples

Figure generated by gnuplot	Figure generated by qtplot (zoom)

Authors

Samy Bengio, Mikaela Keller, Johnny Mariéthoz