It is common practice to standardize (0 mean, 1 stdev) regressors in a regularized setting such as lasso, ridge regression, etc. Pretty much everybody agrees on the fact that this gives a fair treatment of each regressor in terms of how much shrinkage applied to the coefficient of each regressor.
There are of course issues when the regressors consist of boolean regressors and continous regressors at the same time. These issues have also been discussed in this forum.
The angle I am taking towards standardization is different. The standardization of a regressors somehow assumes implicitly that the regressors are at least covariance stationary. If a regressor doesn't have a stationary mean, the 0 mean standardization procedure becomes nothing more than an ad-hoc arithmetic. Since the mean is not converging the standard deviation adjustment also becomes dubious.
Moreover behind the scenes it seems we are doing a density estimation for regressors with a Gaussian distribution assumption, and this is why the boolean/categorical regressors don't seem to cooperate well.
In general, do you think that these concerns are valid? How can we improve the situation?