# Significance test for classification systems

I have several classification systems which I want to compare. The way I evaluate each one of them is using precision, recall and f-score. I want to compare the performance of every two of them using some significance test to find out if the difference in the performance is due to noise or not.

My question is - what would be a good choice for significance test, given the fact that I want to use the f-score values as an input. So far, most of the tests I have seen compare means w.r.t. standard deviation, but I am not sure this is what I need in my case.

-
 This is more of a statistics question than a programming question. I will look into possibly migrating this for you to a more appropriate site. – maple_shaft Feb 5 at 18:28 Do your classifiers return continuous valued "probabilities" between 0/1 like a risk prediction or do they return binary Yes/No predictions? – AdamO Feb 5 at 22:17 What is your utility (cost/loss) function and is it consistent with the classifications you are forcing? Have you read about proper scoring rules? – Frank Harrell Mar 8 at 23:29 I think what the user is asking, is what statical test is appropriate to test the difference between two $F$ scores. Would a Student's $t$-test suffice? Whether done by 10-folds cross validation or 5x2 fold cross validation. – entropy Mar 9 at 6:34