2
$\begingroup$

I have trained a model using caret's train function in R (LASSO logistic regression with glmnet, repeatedcv). The goal is both feature selection and getting the model. With the preferred lambda, LASSO penalized many variables to zero, as expected. I then tried to use the model to predict outcomes with the predict function, that data only includes variables that were not penalized to 0 in the model training. Surprisingly, the predict function returns an error and expects all variables, including those that were non-predictive in feature selection.

How can I completely drop these variables from the model to use it for predictions? Am I missing an obvious point here?

$\endgroup$
4
  • $\begingroup$ Why do you need to drop variables whose parameters are estimated as $0?$ $\endgroup$
    – Dave
    Oct 7, 2020 at 2:01
  • 1
    $\begingroup$ Prediction is just a matrix multiplication with the coefficient vector (assuming you have a Gaussian model, otherwise you also need to apply the inverse link function). So, something like this should work for you: coef <- coef(fit, s = preferred_lambda); cbind(1, newx) %*% as.vector(coef[coef != 0]) $\endgroup$
    – Roland
    Oct 7, 2020 at 6:21
  • $\begingroup$ @Roland Thanks! I feel like I am very close, but am not entirely sure how to implement this in my fit. The fit is from caret (class= 'train'). I can extract the coefs as expected, but am not sure how to subset the fit object using the final coef vector. The goal is to run predict(fit, newdata, type = "prob"). That function indeed already seems to automatically use the bestTune that is stored in the fit (so correctly chooses alpha and lambda), but still expects the irrelevant variables as input. So far I've tried to subset fit$coefnames , but that doesn't do the trick. $\endgroup$
    – Felix
    Oct 8, 2020 at 3:50
  • $\begingroup$ @Dave Because they are otherwise expected as input using predict(), even though their values are not considered in the prediction. $\endgroup$
    – Felix
    Oct 10, 2020 at 22:56

1 Answer 1

0
$\begingroup$

Those features are in the model, just with their coefficients set to zero. Consequently, the software is being quite reasonable in expecting those variables in the data frame for which you want to make predictions.

I see three options.

  1. Supply those variables, knowing the values will not affect the predictions (since the coefficients are zero).

  2. Make up values for those variables (such as all zero), since their values do not affect the predictions. This could be useful if you need to do complex data wrangling to access those values, resulting in either slow performance or irritation to the programmer. This could be useful, too, if you stop collecting data on the variables that do not survive the LASSO estimation.

  3. Do the vector multiplication on your own, outside of the usual prediction method. This will involve matrix multiplication of the data frame with the “surviving” features times the vector of nonzero LASSO-estimated regression coefficients. You’re always allowed to say that the model is something like $\hat y_i=\hat\beta_0+0x_{i1}+\hat\beta_2x_{2i}=\hat\beta_0+\hat\beta_2x_{2i}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.