Timeline for Significance testing of cross-validated classification accuracy: shuffling vs. binomial test
Current License: CC BY-SA 3.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 11, 2014 at 11:56 | vote | accept | amoeba | ||
| Mar 7, 2014 at 18:11 | comment | added | amoeba | @julieth: I did some extra reading and extra simulations, and came to the conclusion that you were right! Please see my updated answer. | |
| Mar 6, 2014 at 23:55 | comment | added | amoeba | @julieth: I agree with cbeleites in that I don't understand how the dependence of training sets is a problem for binomial testing. I saw this sentence in Jiang et al. yesterday and did not really understand it. But see a reply to this question that I just posted: I provided a very simple simulation there and I am wondering if you will think that the dependence of training sets is the issue there. | |
| Mar 6, 2014 at 21:17 | comment | added | julieth | @cbeleites: In the paper I referenced, Jiang et al write: "For LOOCV, however, the test set on which prediction of the ith case has n-2 specimens in common with the training set on which prediction of the jth is based, hence, the number of prediction errors is not binomial". This has been discussed in other places as well. To be clear, I was referring to the dependence of the folds on one-another. | |
| Mar 6, 2014 at 20:51 | comment | added | cbeleites | @julieth: I agree that binomial is not allowed here - based on the "symptoms" amoeba describes. But I'm not yet convinced the dependence of the training sets is the problem. Moreover, I don't really have a good idea what to do :-( | |
| Mar 6, 2014 at 20:37 | comment | added | julieth | @cbeleites: For the current case, it was about why the usual binomial test cannot be used to assess significance and about putting an interval around the binomial proportion. Do you agree that in this case the dependence makes it so that the classical formula cannot be used? Thanks. | |
| Mar 6, 2014 at 13:44 | comment | added | cbeleites | @julieth: whether the dependence of the training sets across the folds (and iterations) is a problem or wanted depends on what exactly you want to measure. If you want to estimate performance for unknown cases for the model built on the whole data set, this is desired behaviour. If you want to estimate performance of models built on $n$ cases of the underlying population, it is a problem. | |
| Mar 6, 2014 at 0:27 | comment | added | julieth | This relates to the dependence I was referring to. The binomial test assumes independence, which you have if you also have an independent validation set. But using cross-validation, the results from one fold are dependent because the training set shared observations in common. | |
| Mar 5, 2014 at 23:16 | comment | added | amoeba | I read your Radmacher et al. 2002 paper. I appreciate your Monte Carlo procedure, but I am also confused because it seems that you can achieve the same conclusions with a MUCH simpler binomial testing. You wrote: "The question is this: is a cross-validated error rate of 3 out of 14 small enough to ensure the class labels for this data set are not artifactual?" Well, 95% binomial CI given 11 correct classifications out of 14 is [0.49, 0.95] which does not exclude 0.5, so the direct answer is no. My question here is precisely this: is permutation test still preferred, and why? | |
| Mar 5, 2014 at 22:54 | comment | added | amoeba | Thank you! It took me some time to find the papers you cite, so I would like to leave the links here: A Paradigm for Class Prediction Using Gene Expression Profiles and Calculating Confidence Intervals for Prediction Error in Microarray Classification Using Resampling. I will read the papers and get back to you after that. | |
| Mar 5, 2014 at 19:19 | history | answered | julieth | CC BY-SA 3.0 |