Confidence curves: an alternative to null hypothesis significance testing for the comparison of classifiers

Daniel Berrar

Research output: Contribution to journalArticlepeer-review

14 Citations (Scopus)


Null hypothesis significance testing is routinely used for comparing the performance of machine learning algorithms. Here, we provide a detailed account of the major underrated problems that this common practice entails. For example, omnibus tests, such as the widely used Friedman test, are not appropriate for the comparison of multiple classifiers over diverse data sets. In contrast to the view that significance tests are essential to a sound and objective interpretation of classification results, our study suggests that no such tests are needed. Instead, greater emphasis should be placed on the magnitude of the performance difference and the investigator’s informed judgment. As an effective tool for this purpose, we propose confidence curves, which depict nested confidence intervals at all levels for the performance difference. These curves enable us to assess the compatibility of an infinite number of null hypotheses with the experimental results. We benchmarked several classifiers on multiple data sets and analyzed the results with both significance tests and confidence curves. Our conclusion is that confidence curves effectively summarize the key information needed for a meaningful interpretation of classification results while avoiding the intrinsic pitfalls of significance tests.

Original languageEnglish
Pages (from-to)911-949
Number of pages39
JournalMachine Learning
Issue number6
Publication statusPublished - 2017 Jun 1
Externally publishedYes


  • Confidence curve
  • Multiple comparisons
  • Performance evaluation
  • Significance test
  • p value

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence


Dive into the research topics of 'Confidence curves: an alternative to null hypothesis significance testing for the comparison of classifiers'. Together they form a unique fingerprint.

Cite this