8
$\begingroup$

I am trying to classify a small dataset (around 500 records) into two classes. I used various methods like SVM, Naive Bayes and k-nn classifier. Now, I would like to set the results from one of the classifiers are my baseline and perform a statistical hypothesis testing. I am not much familiar with this field of statistical testing, and I wonder how to proceed on this.

I have been thinking of setting the SVM classifier as my baseline, but I am not sure how to perform a t-test (or similar) on the data. The input dataset has 10 attributes. Should I use the classification results from two classifiers and do a paired t-test on them? For example, I could take the result from Naive Bayes and perform the paired t-test with the SVM results (which is the baseline). Is this the right approach?

Also, I am confused with the explanation for null and alternate hypothesis. Could someone be willing to give an idea about how to fix the null and alternate hypothesis.

Improve this question
$\endgroup$
3

3 Answers 3

Reset to default
4
$\begingroup$

In general layman's terms (and not just for this problem),

  • Null Hypothesis $H_0$: no change or difference (i.e. the classifiers have the same performance, however you define it)
  • Alternative Hypothesis: there is some sort of difference in performance

For your classifier performance comparison problem, I recommend reading Chapter 6 of Japkowicz & Shah, which goes into detail on how to use significance testing to assess the performance of different classifiers. (Other chapters give more background on classifier comparison - sounds like they might interest you too.)

In your case,

  1. to compare 2 classifiers (on a single domain) you may use a matched-pairs t-test where $t=\frac{\bar{d}}{\bar{\sigma}_d / \sqrt{n}}$, where $\bar{d} = \bar{\text{pm}}(f_1) - \bar{\text{pm}}(f_2)$ is the difference of the means of your performance measures (whatever you choose to use) based on applying the two classifiers $f_1$ and $f_2$, $n$ is the number of trials and $\bar{\sigma}_d$ is the sample standard deviation of the mean difference
  2. to compare multiple classifiers (on a single domain) you may use one-way ANOVA (i.e. an F-test) to check whether there is any difference among multiple means (though it cannot tell which are actually different) and then follow up with post-hoc tests, such as Tukey's Honest Significant Difference test to identify which pairs of classifiers exhibit significant differences.

The book goes into far more detail, so I do recommend reading that chapter.

And in terms of baselines, the tests I've mentioned don't distinguish between a baseline and a non-baseline. This is a good thing, as it gives you flexibility to decide which comparisons you should give more importance to in your analysis. The number of tests you actually do determines whether you should rely on 1. or 2. above.

Improve this answer
$\endgroup$
0
$\begingroup$

One-way ANOVA followed by post hoc tests using Tukey-Kramer method for multiple comparison analysis is one possible approach.

As a result various plots are generated where in each graph, the mean of the classifier performance measure and an interval (95% confidence interval) are clearly represented. Two means are significantly different if their intervals are disjoint, and are not significantly different if their intervals overlap.

In the reference, I have included a research article from our group, where we statistically compare the outputs from different classifiers, and take out the best one.

Reference:

M.L. McHugh, Multiple comparison analysis testing in ANOVA, Biochemia medica 21 (2011) 203–209. http://www.ncbi.nlm.nih.gov/pubmed/22420233

R. Prashanth, S Dutta Roy, PK Mandal, S Ghosh, High-Accuracy Detection of Early Parkinson's Disease through Multimodal Features and Machine Learning, 90 (2016) 13–21. http://www.sciencedirect.com/science/article/pii/S1386505616300326

Improve this answer
$\endgroup$
0
$\begingroup$

I would not do an ANOVA because your experimental design corresponds to a one way within subjects (the algorithms are tested on identical test-set folds). Such an ANOVA has a sphericity assumption which you cannot assume is met by your data.

But this is not a problem anyway. What you really need are those post-hoc tests. They tell you much more than a simple ANOVA ever could. You can perform them without the ANOVA.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.