8
$\begingroup$

This may be perhaps a silly question, but when generating a model with caret and using something like LOOCV or (even more to the point) LGOCV, what is the benefit of splitting data into train and test sets if this is essentially what the cross-validation step does anyway?

I read some of the related questions and they suggested that some of the cross-validation methods (e.g. what is described here at the caret site) are for the purpose of feature selection. But in my case, I'm using randomForest (method = "rf") and kernlab (method = svmRadial), which aren't listed in the group that attempts to purge predictors.

So, my question is if I use something like cross_val <- trainControl(method = "LGOCV", p = 0.8), isn't that the same as training on 80% of my data, testing the resultant model on the remaining 20%, and doing that over and over to get an idea of how well the model is working?

If so, is there any need to split my data into train/test sets?

P.S. I partly ask as I'm conducting models on empirically generated DOE prototypes (think hard goods where we tweak inputs and then use test methods to measure various attributes about the prototype).

As such, I don't have a huge data set with lots of overlapping predictor levels to model from -- we often run one trial at each DOE point of interest since data generation is expensive in this case. Thus, I'd like to use all the data I can for an accurate model, but wanted to check here that I'm not missing something obvious and making a poor model by not splitting things up.


Edit: In response to @topepo's question, I'm modeling physically measured attributes of a compound based on adjusting the chemical inputs of the formula. I can't discuss my actual application, but I'll make up an example based on formulating interior latex paint. I'm running designed experiments where we blend 4-5 chemicals, maybe play with % solids, and an amount of time to heat the polymer solution to adjust the degree of polymerization.

We then might measure rheology, molecular weight, hardness of the paint coating, water resistance, etc.

We have decent replicates of several variables, but few true replicates in the sense that every DOE level was exactly the same. Total data set is ~80 observations and maybe 4-5 are exact repeats. We've conducted 15 different tests, and perhaps 5-6 of them have been done for every single observation. Some of the responses are present for 25-50% of the data.

From here, we'd like to model the effects of our 7 predictors on the output properties and then optimize to target new design spaces that are most likely to give the desired properties.

(Hence my question HERE. Once I have a trained model, it would be nice to do the "reverse" and feed in desired responses to get the best guess at possible input levels to try next).

$\endgroup$
  • $\begingroup$ Validation set and test set are two different things! See stats.stackexchange.com/questions/19048/… and en.wikipedia.org/wiki/Test_set $\endgroup$ – sashkello Jul 23 '13 at 1:44
  • $\begingroup$ @sashkello Thanks for that. This can probably be closed as a duplicate, and somehow I missed that question entirely. One missing bit might be: If I train with data_set1, what do I consider the step performed by LGOCV cross-validation? From my reading I'm assuming 1) caret iterates through tuning parameters on data_set1 and then 2) holds those params fixed and 3) creates a "sub model" using params from #1 for each p = 0.8 sample of data_set1 and tests predictions on the remaining 0.2 to gauge accuracy. Is that a reasonable summary? $\endgroup$ – Hendy Jul 23 '13 at 1:58
  • $\begingroup$ One other missing bit: does leaving out a set for testing help model accuracy at all? It sounds like it just gives an unbiased set for future testing. If data is limited and there are not a lot of overlapping design points, aren't I better off training with the full set and relying on cross-validation for an accuracy estimate? $\endgroup$ – Hendy Jul 23 '13 at 2:01
  • $\begingroup$ With better search strategy, this is anoter helpful one: stats.stackexchange.com/questions/9357/…. Particularly this "I would think that one limiting factor here is how much data you have. Most of the time, we don't even want to split the data into fixed partitions at all, hence CV." I think this gets at my question (which is probably specific to the limited nature of my data (only 80 observations). Seems like a vote for not splitting in these cases? $\endgroup$ – Hendy Jul 23 '13 at 2:06
6
$\begingroup$

My general thoughts:

So when you are evaluating different models, you may want to tune them, try different types of pre-processing etc until you find what you think is a good model. Resampling can help guide you in the right direction during that process.

However, there is still the chance of over-fitting and the odds of this happening is greatly influenced by how much data (and predictors) you have. If you have a little bit of data, there are a few ways to think about this:

  • Use all the data for training since every data point adds significantly to how well the model does.
  • Set aside a small test set as a final check for gross errors due to over-fitting. The chances of over-fitting with a small samples size is not small and gets bigger with the number of samples.

I fall into the second camp but the first isn't wrong at all.

If you have a ton of data then it doesn't really matter much (unless you ave a small event rate).

For you:

You have a DOE. The type of design would help answer the question. Are you trying to interpolate between design points or predict design points that have not been tested so far?

You have one replicate. I fell like random forest is hitting a nail with a sledge hammer and might result in over-fitting. I would try something smoother like an SVM or (gasp) neural network.

Max

$\endgroup$
  • $\begingroup$ I added an update for you above. I've had pretty nice curves (I guess that's subjective) with method = "svmRadial". I hope to evaluate some others, including method = neuralnet, though in past attempts I've found the neural network very sensitive to `set.seed(). In short, we're trying to 1) better understand the relationship between inputs/multiple responses and 2) make best guesses at the most likely design space to yield desired properties so that we can run new, more focused DOEs vs. a more typical expansive/shotgun approach in terms of input min/max levels to try. $\endgroup$ – Hendy Jul 24 '13 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.