# Running tests over samples from cross-validation on multiple datasets. To average or not to average?

Let us say that I am trying to investigate whether we can reliably decode (i.e., predict) some information from some data. The particular scenario is predicting the object a person is seeing from the neural activations in the brain. Basically, it is a classification problem. I run k-fold cross-validation to get k accuracy values. I have multiple subjects (let us say N subjects), and I run the same analysis for each subject; hence, I have k accuracy values for each subject (i.e., in total k*N accuracy values).

I want to test whether these accuracies are significantly above chance. I can run a t-test over all the accuracies (k*N samples in t-test), or I can first average k accuracies for each subject to get N accuracy values, and then run t-tests over these N samples. Which one is the right way? I feel that first averaging and then testing is the right way because the samples you get from k-fold cross-validation are not independent; therefore, without averaging we are being over-confident. Is that right?

Maybe running t-tests over accuracies is not the right way to go about this. Please let me know if you have any other ideas.

You should form the test on the $k$ errors, testing the mean of these $k$ samples under the null distribution. Each fold is a guess at the true fit of the model using a bootstrap-like approach, so naturally this should make sense as we are comparing the model to "random guessing".
• In order to see if the model is statistically significant in general, you test against a null hypothesis, i.e., "random guessing". This can be reformulated as a null model, and thus what you now do is model selection comparing how different the two models are. If you want to do an averaging using CV as your comparison, then what I described above is the best way to do so. You form a $t$-test on one statistic, the mean of the $k$ errors. This is slightly problematic as I mentioned above, but not as much as the other proposed $t$-tests. – Dustin Tran Jan 2 '15 at 21:37