2
$\begingroup$

I work with the book 'Introduction to statistical learning with R', link. I got a question about the cross-validation for ridge and lasso regressions (ellipsis indicate that part of the original code was omitted for brevity. The code can be found here).

x=model.matrix(Salary~.,Hitters)[,-1]
y=Hitters$Salary

# Ridge Regression

library(glmnet)
grid=10^seq(10,-2,length=100)
ridge.mod=glmnet(x,y,alpha=0,lambda=grid)
...

set.seed(1)
train=sample(1:nrow(x), nrow(x)/2)
test=(-train)
y.test=y[test]   
...

set.seed(1)
cv.out=cv.glmnet(x[train,],y[train],alpha=0)

My question is: what is the reason for applying the cross-validation to the train subset of the original data? Doesn't it already include splitting data into the training and test sets? In Chapter 5 Lab the cross-validation is performed on the entire data set.

EDIT. Let me rephrase the question: why would you use the cross-validation on the train subset in the example above as opposed to using it on the entire set like this:

cv.out=cv.glmnet(x,y,alpha=0)

What are the advantage of subsetting if the aim is finding the best model fit?

$\endgroup$
2
  • 2
    $\begingroup$ I believe you are using CV in this context to select appropriate parameters for glmnet. So you would do that on the training data first, but then once you've picked an appropriate parameter, you'd fit glmnet using that parameter, and then use the hold out sample (the test data) to see how the mode performs. $\endgroup$ Feb 2, 2017 at 17:55
  • 1
    $\begingroup$ To repeat @michaeloberst, glmnet is using k fold cross validation on the training data to find the best regularisation parameter by minimising "validation set" error. To then get an unbiased estimate of generalisation you need a hold out set, the test set, or you can use nested cross validation on the whole data set, splitting the data repeatedly into test and "train/validation" , then splitting train data repeatedly into train and validation $\endgroup$
    – seanv507
    Feb 6, 2017 at 6:46

1 Answer 1

2
$\begingroup$

In a single sentence: in order to evaluate your final model on truly unseen data.

Deeper explanation

One big reason to only use your training data while trying different models is to leave some portion of your data totally untouched and unexplored to validate your results on. This way, you can evaluate your final model on completely unseen data.

While the point of cross-validation is to have a better method of evaluating your model than a single train/test split, if you're using cross-validated model training on your full dataset then there technically isn't any truly unseen data by the end, and you might make decisions about which model to go with based on noise in the data that you slowly fit to over the course of many model iterations.

Conclusion

I haven't seen research on this question exactly, so it might not actually be an issue. It's not the worst thing to use your whole dataset in your training if you're using cross-validation, but there may be small changes of not generalizing well to new data—I personally try to leave out a test set that I don't look at until I have my final model. I'd love to see research on the answer.

Note: This answer also discusses this question: Do we need a test set when using k-fold cross-validation?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.