Timeline for Significance testing of cross-validated classification accuracy: shuffling vs. binomial test
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2017 at 12:44 | history | edited | CommunityBot |
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
|
|
| Mar 14, 2014 at 15:43 | comment | added | amoeba | @julieth: One thing I am still confused about though. Taking your example with 5 observations: if instead of classifying the held-out sample we choose to classify a random sample, then the variance will be strictly binomial. Even though trainings sets still have a majority of observations in common. So I am once again not sure that non-independence of the training sets is the correct (or full) explanation of over-dispersion of CV results under shuffling. Any thoughts about that? | |
| Mar 11, 2014 at 17:34 | comment | added | cbeleites | @amoeba: and no, I didn't want to imply that your code is wrong. But I think in order to understand what is going on it would be good to approach also from different corner cases and add "sneak preview" by testing with large independent test sets. I think we also need to be extra careful not to mix up to which situation exactly we want to generalize. Intuitively would not have expected the variance to increase that much, but randomly relabeling should lead to highly unstable models, so increased variance over binomial is to be expected. | |
| Mar 11, 2014 at 16:53 | comment | added | cbeleites | "I wonder how many papers there are out there using binomial confidence intervals and tests" My applied impression is: far less than papers having no thought whatsoever on confidence intervals and reporting 4 digits of sensitivity on a basis of < 20 independent cases... | |
| Mar 11, 2014 at 16:51 | comment | added | cbeleites | @amoeba: yes I'd like to dig deeper into this. The more so, as I'm not yet sure that what Bengio discusses as generalization error is the same kind of generalization that most people think of when discussing generalization error for an application as opposed to the generalization error of an algorithm I suspect it is related to the difference between "a" vs. "the" model. Should we "meet" (email?) outside SX to see what we can do? | |
| Mar 8, 2014 at 0:01 | comment | added | julieth | Consider 5 observations. If we hold out "1", 2-5 make up the training set. If we hold out "2" 1, 3-5 make up the training set. So the training sets have a majority of observations in common and therefore predictions will not be independent. There are many ways to go wrong in cross-val, and many strong opinions out there about what is right and wrong. I commend your detailed study of a question you sought an answer for. | |
| Mar 7, 2014 at 18:14 | history | edited | amoeba | CC BY-SA 3.0 |
short correction
|
| Mar 7, 2014 at 18:10 | comment | added | amoeba | @cbeleites: I added an important update. It seems to me that now that julieth was right, it's the dependence of training sets that screws up binomiality. I would appreciate if we could discuss this further once you have some time, because it seems to be important, and at the moment I have the impression that it contradicts your claims about binomial intervals. | |
| Mar 7, 2014 at 18:08 | history | edited | amoeba | CC BY-SA 3.0 |
updated
|
| Mar 7, 2014 at 8:16 | comment | added | amoeba | @cbeleites: So you think I have a mistake? I will double check and can also ask a colleague of mine to reproduce the simulation in Python. However, it does seem plausible to me. Binomial variance assumes that on each shuffling probability $p$ of correct decoding is 50%. But some shuffles will result in a bit better class separability, some in worse, so $p$ will fluctuate around $p=0.5$ and this will inflate the overall variance. | |
| Mar 7, 2014 at 0:34 | comment | added | cbeleites | Sorry, I don't have time right now to think about this (nor at the weekend) but I think you need to post the actual code. It is also possible that for that question code review is the more suitable stackexchange. | |
| Mar 6, 2014 at 23:47 | history | answered | amoeba | CC BY-SA 3.0 |