Let us say that I am trying to investigate whether we can reliably decode (i.e., predict) some information from some data. The particular scenario is predicting the object a person is seeing from the neural activations in the brain. Basically, it is a classification problem. I run k-fold cross-validation to get k accuracy values. I have multiple subjects (let us say N subjects), and I run the same analysis for each subject; hence, I have k accuracy values for each subject (i.e., in total k*N accuracy values).
I want to test whether these accuracies are significantly above chance. I can run a t-test over all the accuracies (k*N samples in t-test), or I can first average k accuracies for each subject to get N accuracy values, and then run t-tests over these N samples. Which one is the right way? I feel that first averaging and then testing is the right way because the samples you get from k-fold cross-validation are not independent; therefore, without averaging we are being over-confident. Is that right?
Maybe running t-tests over accuracies is not the right way to go about this. Please let me know if you have any other ideas.