If you build a MARS model with cross validation (say 5-fold), do you still need to split the data into training and test and test your model? I thought you are training on four-folds and testing on one-fold of the data by using 5-fold cross validation?
2 Answers
$\begingroup$
$\endgroup$
With MARS, or any other method for that matter, once you have your model you usually want a final validation set. Cross validation is really an assessment of your method and variable choice.
You split the data up so you can get a sense of how the method you're using will perform on a wide variety of data. If you just had one test and training split, you might end up with some idiosyncrasies of the data yielding a too high or too low estimate of out of sample error with your method of choice. However, with cross validation you're getting to use all of the data as both training and test, though not at the same time. Averaging across every round of cross validation gives you and estimate of how your method will perform out of sample for some future as-of-yet unobserved data.
Even with cross validation though, you want to leave out a validation set to "keep you honest" so to speak. Since you can cross validate over and over with different models and values of tuning parameters, it's easy to fall into the trap of thinking that the one with the best out of sample CV error will be the best on a totally new dataset.
$\begingroup$
$\endgroup$
K-fold cross validation is just a process that repeats the split of training and test data k times, each time with a different subset of data used as test. You cannot somehow skip this.
