2
$\begingroup$

I have several classification systems which I want to compare. The way I evaluate each one of them is using precision, recall and f-score. I want to compare the performance of every two of them using some significance test to find out if the difference in the performance is due to noise or not.

My question is - what would be a good choice for significance test, given the fact that I want to use the f-score values as an input. So far, most of the tests I have seen compare means w.r.t. standard deviation, but I am not sure this is what I need in my case.

Improve this question
$\endgroup$
5
  • $\begingroup$ This is more of a statistics question than a programming question. I will look into possibly migrating this for you to a more appropriate site. $\endgroup$
    – maple_shaft
    Commented Feb 5, 2013 at 18:28
  • 1
    $\begingroup$ Do your classifiers return continuous valued "probabilities" between 0/1 like a risk prediction or do they return binary Yes/No predictions? $\endgroup$
    – AdamO
    Commented Feb 5, 2013 at 22:17
  • $\begingroup$ What is your utility (cost/loss) function and is it consistent with the classifications you are forcing? Have you read about proper scoring rules? $\endgroup$
    – Frank Harrell
    Commented Mar 8, 2013 at 23:29
  • 3
    $\begingroup$ I think what the user is asking, is what statical test is appropriate to test the difference between two $F$ scores. Would a Student's $t$-test suffice? Whether done by 10-folds cross validation or 5x2 fold cross validation. $\endgroup$
    – entropy
    Commented Mar 9, 2013 at 6:34
  • 2
    $\begingroup$ Look at this paper. It covers everything you need to know. In short: you should use Friedman test for an overall test between all classifiers, then use paired Wilcoxon signed-rank tests to compare them two-by-two. In the end, you also have to apply a certain Type I correction method to correct for the paired tests - my favourite is the Holm-Bonferroni method. $\endgroup$
    – alesc
    Commented Mar 23, 2015 at 13:26

2 Answers 2

Reset to default
1
$\begingroup$

@Community just brought up this question from the yesteryear of 2013! I think not enough emphasis was given to @alesc answer: The paper Statistical Comparisons of Classifiers over Multiple Data Sets by Demsar has all the answers you need to compare many classifiers.

Improve this answer
$\endgroup$
0
$\begingroup$

You can do this by using the performance of your systems in cross-validation on your training data (say, for example, 5x2 cross-validation, or five repetitions of 2-way) to generate confidence intervals.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ To have sufficient precision you may need 100 repeats of 10-fold cross-validation. Or just use the bootstrap. $\endgroup$
    – Frank Harrell
    Commented Mar 8, 2013 at 23:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.