Statistic test on percentage correct classified by emotion recognition For a potential emotion recognition bachelor-project I was wondering what statistical test I have to perform when I get my results to test whether it's significant. I will be testing which combination of feature extraction and machine learning algorithm will give me the best percentage of correct classified. The results will exist out of Combination A gives ...% classified, Combination B ...%, Combination C ...% and so on. Which statistical test should I use to test whether Combination ... is significantly better than the others and why?
For example: 6 emotions have to be recognized in a database with 100 faces for each emotion (600 total). Every machine learning algorithm will use 2/3 for training and 1/3 for testing. Which face per emotion is in the training set and which one is in the test set is randomly selected every epoch for 100 epochs. The end result is for example: Combination A classified 93.4% correct, Combination B 91.2%, Combination C 86.3% and so on. Which statistical test should I use to test whether Combination A is significantly better than Combination B (and C) and why?
Also is it dichotomous? As the probability of successful selecting the right emotion is 16.67%.
 A: This is a surprisingly tricky issue. If you train and test your 'combinations' only once, statistical testing is straightforward: you can compare each pair of 'combinations' using McNemar's test, (see this question). Then you can correct the p-values for running multiple tests by FDR or Bonferroni, depending how strict you want to be. 
However, you want to average across many splits. This makes the variance of the accuracy statistic difficult to estimate, since the accuracies of different splits are not independent from each other. One way to deal with this problem is to ignore it, treating the 100 accuracies (generated from 100 splits) as 100 independent observations. If you choose this path, you can use Wilcoxon signed-rank test to compare pairs of 'combinations' (and then correct for multiple comparisons for testing several pairs). However, there is a danger of an optimistic bias - your p-values will be probability too small. The extent of the problem will depend on your particular dataset and algorithms.
If you want to avoid this optimistic bias and still test accuracy averaged over different splits, you'll have to look for a more sophisticated approach: Nadeau and Bengio, 2003 was mentioned in this question as an attempt at this problem.
