This may be perhaps a silly question, but when generating a model with caret and using something like
LOOCV or (even more to the point)
LGOCV, what is the benefit of splitting data into train and test sets if this is essentially what the cross-validation step does anyway?
I read some of the related questions and they suggested that some of the cross-validation methods (e.g. what is described here at the caret site) are for the purpose of feature selection. But in my case, I'm using
method = "rf") and
method = svmRadial), which aren't listed in the group that attempts to purge predictors.
So, my question is if I use something like
cross_val <- trainControl(method = "LGOCV", p = 0.8), isn't that the same as training on 80% of my data, testing the resultant model on the remaining 20%, and doing that over and over to get an idea of how well the model is working?
If so, is there any need to split my data into train/test sets?
P.S. I partly ask as I'm conducting models on empirically generated DOE prototypes (think hard goods where we tweak inputs and then use test methods to measure various attributes about the prototype).
As such, I don't have a huge data set with lots of overlapping predictor levels to model from -- we often run one trial at each DOE point of interest since data generation is expensive in this case. Thus, I'd like to use all the data I can for an accurate model, but wanted to check here that I'm not missing something obvious and making a poor model by not splitting things up.
Edit: In response to @topepo's question, I'm modeling physically measured attributes of a compound based on adjusting the chemical inputs of the formula. I can't discuss my actual application, but I'll make up an example based on formulating interior latex paint. I'm running designed experiments where we blend 4-5 chemicals, maybe play with % solids, and an amount of time to heat the polymer solution to adjust the degree of polymerization.
We then might measure rheology, molecular weight, hardness of the paint coating, water resistance, etc.
We have decent replicates of several variables, but few true replicates in the sense that every DOE level was exactly the same. Total data set is ~80 observations and maybe 4-5 are exact repeats. We've conducted 15 different tests, and perhaps 5-6 of them have been done for every single observation. Some of the responses are present for 25-50% of the data.
From here, we'd like to model the effects of our 7 predictors on the output properties and then optimize to target new design spaces that are most likely to give the desired properties.
(Hence my question HERE. Once I have a trained model, it would be nice to do the "reverse" and feed in desired responses to get the best guess at possible input levels to try next).