Timeline for Cross Validation (error generalization) after model selection
Current License: CC BY-SA 2.5
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 3, 2014 at 14:46 | comment | added | tobip | @B_Miner The best reference I know for an algorithmic view on creating nested CV is this (Petersohn, Temporal Video Segmentation, Vogt Verlag, 2010, p. 34). | |
| Jan 12, 2011 at 16:29 | comment | added | user2643 | @user2040: Sorry for the late reply. I created my own software implementation of the nested CV approach. Because my research is related to bioinformatics, I'm planning to submit a description of the software soon to a bioinformatics journal. But it can be used in any research domain. If you're interested in trying it out, please let me know. goldfish1434 at yahoo dot com | |
| Jan 5, 2011 at 7:54 | comment | added | chl | @Dikran I should have give more precision about my study: Yes, the objectives was feature selection (in an $n\ll p$ context, as commonly found in genome-wide association studies). When the outcome is univariate, I still prefer the elasticnet criterion, but anyway it's not the purpose of the question. I like your response (+1). | |
| Jan 4, 2011 at 21:08 | comment | added | Dikran Marsupial | If features vary from fold to fold it means that there isn't enough information to confidently identify the useful features, so I'd view that as an advantage of cross-validation (as just looking at the results from a single model is likely to have over-fit the feature selection criterion and hence be misleading). For problems with many features and few observations, ridge regression often gives better performance, so unless identifying features is a key goal, it is often better not to do any feature selection. | |
| Jan 4, 2011 at 19:09 | comment | added | chl | @user2643 The problem with that approach (which is correct) is that it only yields a single criterion for accuracy (classification) or precision (regression); you won't be able to say "those are the features that are the most interesting ones" since they vary from one fold to the other, as you said. I've been working with genetic data (600k variables) where we used 10-fold CV with embedded feature selection, under a permutation scheme (k=1000, to be comfortable at a 5% level) to assess reliability of the findings. This way, we are able to say: "our model generalizes well or not", nothing more. | |
| Jan 4, 2011 at 18:50 | comment | added | B_Miner | @user2643: Do you have any references to share on how you created the nested CV? Was it along the same lines as the pdf I linked to in my question? Also.....is this data marketing data by chance? | |
| Jan 4, 2011 at 18:25 | history | answered | user2643 | CC BY-SA 2.5 |