Timeline for R/caret: train and test sets vs. cross-validation?
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2017 at 12:44 | history | edited | CommunityBot |
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
|
|
| Jul 24, 2013 at 22:46 | history | edited | Hendy | CC BY-SA 3.0 |
added more information on the type of modeling being conducted
|
| Jul 23, 2013 at 11:26 | answer | added | topepo | timeline score: 8 | |
| Jul 23, 2013 at 2:06 | comment | added | Hendy | With better search strategy, this is anoter helpful one: stats.stackexchange.com/questions/9357/…. Particularly this "I would think that one limiting factor here is how much data you have. Most of the time, we don't even want to split the data into fixed partitions at all, hence CV." I think this gets at my question (which is probably specific to the limited nature of my data (only 80 observations). Seems like a vote for not splitting in these cases? | |
| Jul 23, 2013 at 2:01 | comment | added | Hendy | One other missing bit: does leaving out a set for testing help model accuracy at all? It sounds like it just gives an unbiased set for future testing. If data is limited and there are not a lot of overlapping design points, aren't I better off training with the full set and relying on cross-validation for an accuracy estimate? | |
| Jul 23, 2013 at 1:58 | comment | added | Hendy |
@sashkello Thanks for that. This can probably be closed as a duplicate, and somehow I missed that question entirely. One missing bit might be: If I train with data_set1, what do I consider the step performed by LGOCV cross-validation? From my reading I'm assuming 1) caret iterates through tuning parameters on data_set1 and then 2) holds those params fixed and 3) creates a "sub model" using params from #1 for each p = 0.8 sample of data_set1 and tests predictions on the remaining 0.2 to gauge accuracy. Is that a reasonable summary?
|
|
| Jul 23, 2013 at 1:44 | comment | added | sashkello | Validation set and test set are two different things! See stats.stackexchange.com/questions/19048/… and en.wikipedia.org/wiki/Test_set | |
| Jul 22, 2013 at 22:08 | history | asked | Hendy | CC BY-SA 3.0 |