0
$\begingroup$

I have to use p-value test to compare two classifiers. I am getting the error vectors of 2 hypothesis and I am using their difference as an input in ttest function of MATLAB

[H,p] = ttest(error,'Tail','both');

I am getting H as 0 and p as 0.34 As far as I read, this means that we cannot reject null hypothesis in this case. I am not sure I don't know how to interpret that? Can anybody help? And again what will one tailed test do in this case?

Improve this question
$\endgroup$
2
  • $\begingroup$ It is unclear how you are comparing them. Are you using multiple data sets / cross-validation / ...? $\endgroup$
    – Marc Claesen
    Commented Nov 10, 2014 at 10:49
  • $\begingroup$ my dataset is the same. I am using two classifiers to get two different error vectors $\endgroup$
    – KingJames
    Commented Nov 10, 2014 at 17:45

2 Answers 2

Reset to default
1
$\begingroup$

The $t$-test is not a very good choice to compare classifiers, since their performance is not normally distributed. I suggest considering the Wilcoxon signed-ranks test instead.

For a thorough read on this topic please refer to this paper (available here):

Demšar, Janez. "Statistical comparisons of classifiers over multiple data sets." The Journal of Machine Learning Research 7 (2006): 1-30.

The above paper lists multiple arguments against using the $t$-test for classifier comparison.

Improve this answer
$\endgroup$
4
  • $\begingroup$ is there any update on the answer after knowing it is the same dataset and using AUC as the evaluation in cross-validation? $\endgroup$
    – RockTheStar
    Commented May 25, 2016 at 23:50
  • $\begingroup$ The t test does not assume that the sampling distributions are normally distributed. $\endgroup$
    – Stephan Kolassa
    Commented May 10, 2021 at 9:01
  • $\begingroup$ @StephanKolassa That link seems to argue that the sampling distributions are normal (ish), just that original populations don’t have to be normal (ish). $\endgroup$
    – Dave
    Commented May 10, 2021 at 10:25
  • $\begingroup$ @Dave: we may be talking about different things here. The underlying distribution do not need to be normal (like classifier performance here, or the uniforms I use in my simulations in the linked answer). What is (asymptotically) normal is parameter estimates, like means (again, see my answer at the link). But means are precisely what t tests compare. $\endgroup$
    – Stephan Kolassa
    Commented May 10, 2021 at 10:28
0
$\begingroup$

In the example you provided of the ttest, the hypothesis is whether your vector "error" came from a Gaussian distribution with null mean (if you use only one input), or in other words, if both classifiers provide the same accuracy. If you use 2 inputs than you can also specify the Tail. If you choose 'Tail','Right' (ttest(classifier1,classifier2,'Tail','Right')), then your hypothesis is that classifier1 is better than classifier2, and if 'Tail','Left' classifier2 is better than classifier1. In all cases, the hypothesis is true if h=1 (at a pval that you defined: typically pval=0.05).

Take into consideration however that ttest is a parametric test and is not be suitable to all cases. You should first perform a Chi-square test on each variable: h = chi2gof(classifier). If both tests fail (h=0), than your data is normal (gaussian) and you can use the ttest, otherwise you should use a non-parametric test such as : p = ranksum(classifier1,classifier2). The interpretation is the same.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.