Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I have several classification systems which I want to compare. The way I evaluate each one of them is using precision, recall and f-score. I want to compare the performance of every two of them using some significance test to find out if the difference in the performance is due to noise or not.

My question is - what would be a good choice for significance test, given the fact that I want to use the f-score values as an input. So far, most of the tests I have seen compare means w.r.t. standard deviation, but I am not sure this is what I need in my case.

share|improve this question
This is more of a statistics question than a programming question. I will look into possibly migrating this for you to a more appropriate site. – maple_shaft Feb 5 at 18:28
Do your classifiers return continuous valued "probabilities" between 0/1 like a risk prediction or do they return binary Yes/No predictions? – AdamO Feb 5 at 22:17
What is your utility (cost/loss) function and is it consistent with the classifications you are forcing? Have you read about proper scoring rules? – Frank Harrell Mar 8 at 23:29
I think what the user is asking, is what statical test is appropriate to test the difference between two $F$ scores. Would a Student's $t$-test suffice? Whether done by 10-folds cross validation or 5x2 fold cross validation. – entropy Mar 9 at 6:34

migrated from programmers.stackexchange.com Feb 5 at 18:57

1 Answer

up vote 0 down vote

You can do this by using the performance of your systems in cross-validation on your training data (say, for example, 5x2 cross-validation, or five repetitions of 2-way) to generate confidence intervals.

share|improve this answer
To have sufficient precision you may need 100 repeats of 10-fold cross-validation. Or just use the bootstrap. – Frank Harrell Mar 8 at 23:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.