I am running ridge regression on the Boston dataset. There are many write-ups online for how to do ridge regression.
I will write up the two methods and then pose my question
Initialize with the dataset
First method: According to the Stanford Open course on statistics
library('glmnet') library('dplyr') #initialize data matrix for glmnnet z <- colnames(BostonHousing) z <- z[-10] x <- BostonHousing %>% select(one_of(z)) %>% data.matrix() #create ridge regression with every possible lambda fit <- glmnet(x,BostonHousing$medv, alpha=0) #use 10-fold cross validation to choose the lambda with the lowest MSE cv_fit <- cv.glmnet(x,BostonHousing$medv, alpha = 0) #create a ridge regression model with the best lambda fit <- cv_fit$glmnet.fit #calculate training MSE for best ridge regression model min(cv_fit$cvm)
Second method: According to this tutorial
library('caret') #take a random sample of half of the data split <- createDataPartition(y=BostonHousing$medv, p = 0.5, list = FALSE) #create training and test sets train <- BostonHousing[split,] test <- BostonHousing[-split,] #calculate ridge regression on every lambda with the training set ridge <- train(medv ~., data = train, method='ridge', lambda = 4,preProcess=c('scale', 'center')) #use the model to predict values of the test set ridge.pred <- predict(ridge, test) #mse for the test error mean(ridge.pred - test$medv)^2 #select lambda fitControl <- trainControl(method = "cv", number = 10) lambdaGrid <- expand.grid(lambda = 10^seq(10, -2, length=100)) #do ridge regression with the best lambda ridge <- train(medv~., data = train, method='ridge', trControl = fitControl, # tuneGrid = lambdaGrid preProcess=c('center', 'scale') ) #predict the test set using the model from the training set ridge.pred <- predict(ridge, test) #calculate test mse sqrt(mean(ridge.pred - test$medv)^2)
I have a few questions, I hope that's alright.
1- Assuming I use the first method, can I estimate the test error of the ridge model with k-fold cross validation?
It only gives me the training error and I'd like to approximate test error.
2- The second approach uses a validation set. Is that desirable in situations with small sample sizes?
The BostonHousing data is 506 rows by 14 variables.
3- Here is the output in the second method
ridge Ridge Regression 254 samples 10 predictor Pre-processing: centered (10), scaled (10) Resampling: Cross-Validated (10 fold) Summary of sample sizes: 230, 229, 228, 229, 229, 229, ... Resampling results across tuning parameters: lambda RMSE Rsquared MAE 0e+00 0.5963179 0.6835195 0.4131819 1e-04 0.5963073 0.6835296 0.4131761 1e-01 0.5920124 0.6891727 0.4120725 RMSE was used to select the optimal model using the smallest value. The final value used for the model was lambda = 0.1.
Why is ridge regression using resampling? How did they get a lambda of
0.1 when the first method got a lambda of