Are regularization techniques model specific?

Question

When we learn about regularization techniques like LASSO, Ridge, etc... it is often taught alongside regression. When performing LASSO or other penalty based regularization techniques, do the same assumptions for a linear regression need to hold before we do LASSO? I.e. Y is linear in coefficients, Y | X is normally distributed, iid, etc...

Or can we run LASSO on the raw data and once we have the reduced feature set, then transform the features so that they fit the assumptions of a linear regression.

I would imagine, for large predictor sets, it would be tedious to check each feature before hand. But at the same time, if the features are NOT pre-processed to fit the assumptions of a linear model, the coefficient estimates wouldn't be accurate - and thus I don't know how accurate our LASSO results would be. Thanks!

Answer 1

Yes, the assumptions apply. But why are you transforming variables in order to make the model fit the data? Instead of changing the data, change the model.

Quantile regression and robust regression do not make assumptions about the distribution of residuals.

On the other hand, if you have nonlinear relationships, you will have to model those - whether you use LASSO or ridge and whatever kind of regression model you choose. But you should want to model them -- they could be very important and interesting.

Answer 2

Ridge regression is defined in terms of applying $L_2$ penalty to the weights of the model

$$ \operatorname{arg\,min}_\boldsymbol{\beta} \| y - \mathbf{X}\boldsymbol{\beta} \|^2_2 + \lambda\| \boldsymbol{\beta} \|^2_2 $$

LASSO uses $L_1$ penalty instead, dropout is applied to neural network weights as well, etc. Regularization is about making models less complex, not the data. Weights are part of the model and make sense only together with other weights, for example, you couldn't "copy and paste" weights between different regression models. So the fact that regularization has zeroed-out some weights for one model, doesn't need to mean that those features would not be important for other model.

On another hand, there are model agnostic feature-selection algorithms that can be applied to the data to select the "best" set of features that can be used with different algorithms. The purpose of such algorithms it to find such "best" features, but different algorithms may select different features and you never have guarantee that the selected features are "ultimately best".