Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

This question already has an answer here:

I'm working on a classification problem and I have a very high F1 baseline of 85%. I have trained three classification models and I want to know which one is the best. How I can do so?

I tried two ways:

  1. To compare each model against the baseline using paired t-test. So I have tests like:

    baseline vs. model 1 | baseline vs. model 2  | baseline vs. model 3
    

    That tells me that only model 1 is significantly higher than the baseline and so I concluded that model 1 is the best. Is this a valid methodology given that usually classification models are compared against baselines?

  2. To compare all models in one fell swoop with one-way ANOVA. So entered the information of modals 1-3 AND the baseline with gave me a p-value of 0.02 indicating that there is a difference in means. Yet, with a post pairwise test, there is no significance between any of the pairs.

Which method is the correct one?

share|improve this question

migrated from stackoverflow.com Jun 13 '13 at 14:02

This question came from our site for professional and enthusiast programmers.

marked as duplicate by COOLSerdash, Nick Cox, Scortchi, chl Jul 10 '13 at 7:38

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

    
If you're comparing the performance of various classifiers wouldn't it be better to use something like the receiver operating characteristic (ROC) curve? –  Manetheran Jun 11 '13 at 6:11
    
But not all my classifiers are binary. –  Sabba Jun 11 '13 at 15:23

1 Answer 1

up vote 0 down vote accepted

I believe both approaches are valid.

When you run simple pairwise T-tests you are being more sensible for differences and not considering that you will run other tests.

On the other hand, the post hoc tests are more conservative, but sometimes can be a little bit controversial.

Suppose you are running a post hoc tests among 10 groups and there is no difference (at some level of confidence). Weeks later, you decide to run the same test, but only with 5 groups and now you have a difference. What can be inferred in this experiment, if nothing changes?

When you have strong evidence of difference, both approaches should indicate the same and it is done. When you have small fluctuations and "apparently" some divergences between approaches you should consider other signals like: which model the hypotheses fits better, which one is more robust for noise, etc.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.