2
$\begingroup$

Let us say that I am trying to investigate whether we can reliably decode (i.e., predict) some information from some data. The particular scenario is predicting the object a person is seeing from the neural activations in the brain. Basically, it is a classification problem. I run k-fold cross-validation to get k accuracy values. I have multiple subjects (let us say N subjects), and I run the same analysis for each subject; hence, I have k accuracy values for each subject (i.e., in total k*N accuracy values).

I want to test whether these accuracies are significantly above chance. I can run a t-test over all the accuracies (k*N samples in t-test), or I can first average k accuracies for each subject to get N accuracy values, and then run t-tests over these N samples. Which one is the right way? I feel that first averaging and then testing is the right way because the samples you get from k-fold cross-validation are not independent; therefore, without averaging we are being over-confident. Is that right?

Maybe running t-tests over accuracies is not the right way to go about this. Please let me know if you have any other ideas.

$\endgroup$
-1
$\begingroup$

You should form the test on the $k$ errors, testing the mean of these $k$ samples under the null distribution. Each fold is a guess at the true fit of the model using a bootstrap-like approach, so naturally this should make sense as we are comparing the model to "random guessing".

There are still some suspicions to this however, as sample reusage causes the individual scores to not be independent.

More generally, this problem relies on theory of statistical significance for model selection. For a Bayesian treatment, see BIC and/or Bayes factor, which will be more robust to problems such as sample size.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Dustin, I am not sure you answered my question. I can run a t-test separately on each subject (k accuracies); however, that would tell me if the model works for a particular subject. I want to test if the model works in general, i.e., across subjects. For your point about BIC and Bayes factor, I know them. But I don't see how they are relevant to my question. $\endgroup$ – goker Dec 30 '14 at 23:44
  • $\begingroup$ In order to see if the model is statistically significant in general, you test against a null hypothesis, i.e., "random guessing". This can be reformulated as a null model, and thus what you now do is model selection comparing how different the two models are. If you want to do an averaging using CV as your comparison, then what I described above is the best way to do so. You form a $t$-test on one statistic, the mean of the $k$ errors. This is slightly problematic as I mentioned above, but not as much as the other proposed $t$-tests. $\endgroup$ – Dustin Tran Jan 2 '15 at 21:37
  • $\begingroup$ Dustin, I am sorry but you have still not answered my question. Maybe the question is not clear. What I am asking is simply this: if we have k accuracies for each of the N subjects and I want to test if the model works in general (i.e., if the accuracies are over chance), there are two ways we can use a t-test here: 1) run a single t-test over k*N accuracies 2) take average of k accuracies for each subject, hence getting N values in total, and run a t-test over these N values. My question is which one is the better way. $\endgroup$ – goker Jan 3 '15 at 23:37
  • $\begingroup$ Is this a homework question? As I've said, they're both bad. If you're looking for the better of two evils, you can use the same reasoning applied above to try to deduce this. $\endgroup$ – Dustin Tran Jan 4 '15 at 21:10
  • $\begingroup$ NO, this is not a homework question. Unfortunately, you have not added anything to what I said in the question; I have already said the samples are not independent so it seems to be a better idea to run t-tests after averaging. If they are both bad, what is a better way? I know what BIC and Bayes factor are. If you have any ideas about using them here in this scenario, please share them. I don't see how they are immediately applicable. $\endgroup$ – goker Jan 10 '15 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.