1
$\begingroup$

For a potential emotion recognition bachelor-project I was wondering what statistical test I have to perform when I get my results to test whether it's significant. I will be testing which combination of feature extraction and machine learning algorithm will give me the best percentage of correct classified. The results will exist out of Combination A gives ...% classified, Combination B ...%, Combination C ...% and so on. Which statistical test should I use to test whether Combination ... is significantly better than the others and why?

For example: 6 emotions have to be recognized in a database with 100 faces for each emotion (600 total). Every machine learning algorithm will use 2/3 for training and 1/3 for testing. Which face per emotion is in the training set and which one is in the test set is randomly selected every epoch for 100 epochs. The end result is for example: Combination A classified 93.4% correct, Combination B 91.2%, Combination C 86.3% and so on. Which statistical test should I use to test whether Combination A is significantly better than Combination B (and C) and why?

Also is it dichotomous? As the probability of successful selecting the right emotion is 16.67%.

Improve this question
$\endgroup$
2
  • $\begingroup$ Can you clarify 'epoch'? Do you split the data to training and test set 100 times and average the results? $\endgroup$
    – Trisoloriansunscreen
    Commented Mar 10, 2014 at 13:31
  • $\begingroup$ Yes every round of training (epoch) the data will be split in 2/3 for training and 1/3 for testing. The next epoch the faces will again be randomly split in 2/3 training and 1/3 testing. The result is indeed the average % of correct classified over 100 times. $\endgroup$
    – NumesSanguis
    Commented Mar 10, 2014 at 14:12

1 Answer 1

Reset to default
1
$\begingroup$

This is a surprisingly tricky issue. If you train and test your 'combinations' only once, statistical testing is straightforward: you can compare each pair of 'combinations' using McNemar's test, (see this question). Then you can correct the p-values for running multiple tests by FDR or Bonferroni, depending how strict you want to be.

However, you want to average across many splits. This makes the variance of the accuracy statistic difficult to estimate, since the accuracies of different splits are not independent from each other. One way to deal with this problem is to ignore it, treating the 100 accuracies (generated from 100 splits) as 100 independent observations. If you choose this path, you can use Wilcoxon signed-rank test to compare pairs of 'combinations' (and then correct for multiple comparisons for testing several pairs). However, there is a danger of an optimistic bias - your p-values will be probability too small. The extent of the problem will depend on your particular dataset and algorithms.

If you want to avoid this optimistic bias and still test accuracy averaged over different splits, you'll have to look for a more sophisticated approach: Nadeau and Bengio, 2003 was mentioned in this question as an attempt at this problem.

Improve this answer
$\endgroup$
2
  • $\begingroup$ My Research Methodology teacher made a comment that she would use ROC for this problem. I still have to hear her out on the details and will report back here then, but any input on this? Why this would be better or worse than the ones you proposed? $\endgroup$
    – NumesSanguis
    Commented Mar 16, 2014 at 23:51
  • $\begingroup$ ROC based measures are better since they don't involve a binary decision rule and are not biased by differences in the occurrence of different categories. Yet, as far as I know, the sampling distributions of these statistics are not simpler to obtain than the sampling distribution of accuracy / error rate, so that won't solve the issue caused by K-fold cross-validation I described above. $\endgroup$
    – Trisoloriansunscreen
    Commented Mar 22, 2014 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.