# Standardization for Regularized Regression

It is common practice to standardize (0 mean, 1 stdev) regressors in a regularized setting such as lasso, ridge regression, etc. Pretty much everybody agrees on the fact that this gives a fair treatment of each regressor in terms of how much shrinkage applied to the coefficient of each regressor.

There are of course issues when the regressors consist of boolean regressors and continous regressors at the same time. These issues have also been discussed in this forum.

The angle I am taking towards standardization is different. The standardization of a regressors somehow assumes implicitly that the regressors are at least covariance stationary. If a regressor doesn't have a stationary mean, the 0 mean standardization procedure becomes nothing more than an ad-hoc arithmetic. Since the mean is not converging the standard deviation adjustment also becomes dubious.

Moreover behind the scenes it seems we are doing a density estimation for regressors with a Gaussian distribution assumption, and this is why the boolean/categorical regressors don't seem to cooperate well.

In general, do you think that these concerns are valid? How can we improve the situation?

• Just to see if I get your question (especially regarding the non-stationary variable I am not sure what kind of model you have in mind and why shifting the mean would be problematic, a contrast, why OLS would work, as comparison would be helpful. Why is this non-stationary specifically a problem for regularization): 1) How about not penalizing the intercept to solve your first concern? 2) And for your second concern use a different model, with no Gaussian distribution assumption, for instance a penalized logistic regression? Oct 20 '17 at 14:38
• @MartijnWeterings I am not penalizing the intercept. In cointegration models explanatory variables are non-stationary. Gaussianity issue is not related to the error, it is related to the mean estimation for the predictors. Oct 20 '17 at 14:44
• Cagdas, I suspect that you might be having an interesting problem, but could you elaborate a bit more on it and be more specific in your question? For instance... "Gaussianity is related to the mean estimation for the predictors" I do not get what that means. Can you refer to some standard work or formulation of your co-integration model? Maybe provide some mathematical expressions? Oct 20 '17 at 15:26
• As others have said, I think you need to be more clear, at least by providing an explicit form for your model and/or references that clarify your context. The use of "stationary", for example, I find especially confusing, as traditionally that's a term that is used to refer to properties of stochastic processes over time, and it isn't straightforward to me what it would mean to call a mean "stationary" without more context. (For example, if I google "stationary mean regularized regression", THIS thread is the first result, so it certainly does not appear to be common nomenclature...) Oct 20 '17 at 18:26