I want to make sure I am grasping Cross Validation (k-fold or single validation set) correctly in a high-level sense. The idea is to first split the data into 3 sets, Train/Validation/Test. I train multiple models on the train set using various lambda values (lets assume L1 Regularization). This results in various learned coefficient values for each lambda value that I trained with. Check.
Now, I take the trained weights and predict on the validation set (without a regularization term) and find out which lambda-computed coefficients give me the lowest error on my validation set. Check.
After finding out which weights worked best on the validation set, I now take these coefficients/weights and use them on the testing set.
My confusion is simply the following: Do we use the regularization term in our models for training, validation or test error assessment? OR is the regularization term purely kept in the training data for obtaining the penalized coefficient estimates? If we want to assess training error on its own (for comparisons) do we keep the regularization term in as well or do we take the best validated coefficients and fit on the training data without a regularization term (after we have used the training set to find the best penalized weights). In other words, is the training set simply a vehicle for obtaining regularized weights such that we can use those weights to fit/predict and assess Training/Validation/Testing later?
Below is some pseudo R Code to confirm my process using k-fold CV. The idea is to use
cv.glmnet to obtain optimal lambda via different training/val folds. After obtaining this
lambda I use it to fit on my training set (which provides me the penalized coefficients/weights that I want to use down the road). I then use these lambda-computed weights on non-regularized test set. Is this correct?
If I were to just use single set validation, I would do the same except instead of the k-fold method, I would just compute a bunch of lambda models using the training set, then use the learned weights from each of the models and see how well it does on the non-regularized validation set. Given the best validation set error performance, I would then choose those lambda-computed weights on the non-regularized test set. Correct?
library(caret) library(glmnet) # using standardized data for Penalized models X_train <- model.matrix(y~., Train_set_standardized)[,-1] # remove intercept term y_train <- Train_set_standardized$y set.seed(123) CV_for_lambda_LASSO <- cv.glmnet(X_train, y_train, alpha = 1, nfolds = 10) # find best lambda for LASSO CV_for_lambda_LASSO$lambda.min # fit on Train set to get the best coefficients for testing LASSO_fit <- glmnet(X_train, y_train, alpha = 1, lambda = CV_for_lambda_LASSO$lambda.min) coef(LASSO_fit) # nothing penalized to zero Train_LASSO_fit <- as.vector(predict(LASSO_fit, X_train)) # get fitted training values # predict on test set X_test <- model.matrix(y~., Test_set_standardized)[,-1] Test_LASSO_pred <- as.vector(predict(LASSO_fit, X_test))