1
$\begingroup$

I was doing hyperparameter tuning of the alpha variable in sklearn's Lasso (using cross-validation). The optimal alpha that was obtained from it is 0. This means no regularization is needed and it's just a linear regressor. I was wondering what could this implicate regarding the data I'm using and my target.

Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

That result shows that you are not overfitting. This is likely to be the case if you have a lot of training data and few parameters. Or alternatively if the process you are modelling really is a linear one. In either case, this is how linear regression should be done.

Linear regression has the merits that it is fast to calculate, and often that it is familiar to the people who will use the model. But it is used too often where there is no real reason to expect that the true process is linear, or with a large number of predictors with no real mechanistic justification for thinking that they are predictive. In either case, Lasso can mitigate the situation a bit, and help make a less ridiculous model. Your result seems to show that you are not running into these problems.

$\endgroup$
3
  • 1
    $\begingroup$ I don't think it necessarily means that your model is not over-fitting, just that it isn't improved by the form of regularisation used by LASSO. It could also mean that your search for the optimal value for the regularisation parameter did not find it. If the problem is non-linear then LASSO is not the answer, the answer is to use a non-linear model. $\endgroup$ Jan 13, 2023 at 22:38
  • 1
    $\begingroup$ Regarding your third sentence: Even if the data generating process is actually linear and you are fitting the true model (the correct functional form and the relevant variables, but unknown parameters), nonzero shrinkage will likely (I think necessarily, unless there is only one regressor) be optimal. But it is possible that that nonzero amount is very close to zero. $\endgroup$ Jan 14, 2023 at 11:23
  • $\begingroup$ Good point Richard, I stand corrected. $\endgroup$ Jan 16, 2023 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.