Why do the results of LASSO regression differ after removing uninformative variables in glmnet? I am researching therapy response of melanoma patients based on a number of approximately 80 features with a very small sample size of 60 patients. To eliminate features that do not contribute to the model, I am using train from the caret package and glmnet with alpha = 1 to perform LASSO regression. This leads to 4 features being kept in the model and an AUC of 0.72. Out of curiosity I fit the model again using only these 4 features and I get an AUC of 0.85. I thought that by using LASSO regression uninformative variables were set to zero and removed from the model so the results should be the same as when just fitting the 4 remaining features. Is there something I am missing about this approach?
cctrl1 <- trainControl(method="cv", number=10, returnResamp="all",
                       classProbs=TRUE, summaryFunction=twoClassSummary)

# create model matrix with all features
x <- model.matrix(Responder ~ ,data=df)
x <- x[,-1]

set.seed(849)
test_class_cv_model <- train(x, df$Responder, method = "glmnet", 
                             trControl = cctrl1,metric = "ROC",
                             tuneGrid = expand.grid(alpha = 1,
                                                    lambda = seq(0.001,0.2,by = 0.001)))

# extracting maximum AUC
max(test_class_cv_model$results$ROC)
[1] 0.7180556

# identify remaining features
coef(test_class_cv_model$finalModel, test_class_cv_model$finalModel$lambdaOpt)



# generate model matrix with the 4 remaining features
xfeat <- model.matrix(Responder ~ F1 + F2 + F3 + F4,data=df)
xfeat <- xfeat[,-1]

set.seed(849)
test_class_cv_model_feat <- train(xfeat, df$Responder, method = "glmnet", 
                             trControl = cctrl1,metric = "ROC",
                             tuneGrid = expand.grid(alpha = 1,
                                                    lambda = seq(0.001,0.2,by = 0.001)))

# extracting maximum AUC
max(test_class_cv_model_feat$results$ROC)
[1] 0.8472222
```

 A: You’re using different loss functions. Of course they’re different. When you do the LASSO, the variables that get set to zero contribute to the loss function. When you do OLS on the “surviving” variables, the loss function only ever sees those variables.
In LASSO, the loss function sees the "dead" variables and has an opportunity to keep them from having their coefficients estimated as nonzero. Once you eliminate those variables, however, then you are essentially forcing those features to have coefficients of zero no matter what.
A: Define a covariate matrix $X \in \mathbb{R}^{n \times p}$ with columns $\{x_j\}$, where there are $n$ observations of $p$ covariates. Define the response $y \in \mathbb{R}^p$. Suppose we are interested in recovering the linear coefficients $\beta$ of $y$ when regressed on $\{x_j\}$.
The Lasso estimator with tuning parameter $\lambda \geq 0$ is defined as: $$\hat\beta_\lambda = \arg\min_{\beta} \|y-X\beta\|_2^2 + \lambda \|\beta\|_1.$$ Define the associated support as $\hat{S}_{\lambda} = \{ j \in \{1, \dots, p\} \, : \, \hat\beta_{\lambda, j} \neq 0 \}$, the covariate indices whose estimated coefficient $\hat\beta_{\lambda, j}$ is nonzero.
Let us define another estimator which performs the lasso only on the support $\hat{S}_\lambda$, ignoring the coefficients not selected by the Lasso: $$\hat\alpha_\lambda = \arg\min_{\alpha} \left\|y- \sum_{j \in \hat{S}_\lambda} x_j \alpha_j \right\|_2^2 + \lambda \sum_{j \in \hat{S}_\lambda} |\alpha_j|.$$ An interesting identity holds: $\hat{\beta}_{\lambda, j} = \hat{\alpha}_{\lambda, j}$ whenever $j \in \hat{S}_\lambda$. This says that the coefficients agree on $\hat{S}_\lambda$ regardless of whether the covariates in $\hat{S}_\lambda^C$ are included in the regression. Why is this true? It follows from a one line proof: the Lasso finds the coefficient estimates which minimize its objective, and the objective for $\hat\alpha_\lambda$ is the same.
This shows that Dave's argument concerning the covariates $j \not\in \hat{S}_\lambda$ being included in the objective must be wrong. However, it also reveals the real reason. The coefficients $\hat\alpha_\lambda$ are being penalized! The term ``$\lambda \sum_{j \in \hat{S}_\lambda} |\alpha_j|$'' induces shrinkage whenever $\lambda > 0$. It makes sense that these coefficients $\hat\alpha_\lambda$ disagree with the OLS coefficients on $\hat{S}_\lambda$. (Note, this latter method is called the relaxed lasso.)
A: The training of the LASSO model is not only selecting features but also restricting the magnitude of the coefficients.
It can be (and often is) that the LASSO model finds some optimum that is restricted by the addition of non-informative features to the model (features that would fit the noise and generalize bad and perform bad). If the parameter for the penalty term would be further decreased then more (bad) features will be included.
(So it can be that your LASSO model is 'wrongly' restricting the magnitude of the coefficients for the included correct features, because it has to keep the number of features low)
Without those other features you will get that the LASSO model is able place the optimum at a different place and increase the magnitude of the coefficients of the four features (without those other features being included).
A similar thing happens when you perform OLS with the features selected by LASSO. You will in general get larger coefficients that may potentially give a better model.
