1
$\begingroup$

It is common practice to standardize (0 mean, 1 stdev) regressors in a regularized setting such as lasso, ridge regression, etc. Pretty much everybody agrees on the fact that this gives a fair treatment of each regressor in terms of how much shrinkage applied to the coefficient of each regressor.

There are of course issues when the regressors consist of boolean regressors and continous regressors at the same time. These issues have also been discussed in this forum.

The angle I am taking towards standardization is different. The standardization of a regressors somehow assumes implicitly that the regressors are at least covariance stationary. If a regressor doesn't have a stationary mean, the 0 mean standardization procedure becomes nothing more than an ad-hoc arithmetic. Since the mean is not converging the standard deviation adjustment also becomes dubious.

Moreover behind the scenes it seems we are doing a density estimation for regressors with a Gaussian distribution assumption, and this is why the boolean/categorical regressors don't seem to cooperate well.

In general, do you think that these concerns are valid? How can we improve the situation?

$\endgroup$
4
  • 1
    $\begingroup$ Just to see if I get your question (especially regarding the non-stationary variable I am not sure what kind of model you have in mind and why shifting the mean would be problematic, a contrast, why OLS would work, as comparison would be helpful. Why is this non-stationary specifically a problem for regularization): 1) How about not penalizing the intercept to solve your first concern? 2) And for your second concern use a different model, with no Gaussian distribution assumption, for instance a penalized logistic regression? $\endgroup$ Oct 20 '17 at 14:38
  • $\begingroup$ @MartijnWeterings I am not penalizing the intercept. In cointegration models explanatory variables are non-stationary. Gaussianity issue is not related to the error, it is related to the mean estimation for the predictors. $\endgroup$ Oct 20 '17 at 14:44
  • 1
    $\begingroup$ Cagdas, I suspect that you might be having an interesting problem, but could you elaborate a bit more on it and be more specific in your question? For instance... "Gaussianity is related to the mean estimation for the predictors" I do not get what that means. Can you refer to some standard work or formulation of your co-integration model? Maybe provide some mathematical expressions? $\endgroup$ Oct 20 '17 at 15:26
  • 2
    $\begingroup$ As others have said, I think you need to be more clear, at least by providing an explicit form for your model and/or references that clarify your context. The use of "stationary", for example, I find especially confusing, as traditionally that's a term that is used to refer to properties of stochastic processes over time, and it isn't straightforward to me what it would mean to call a mean "stationary" without more context. (For example, if I google "stationary mean regularized regression", THIS thread is the first result, so it certainly does not appear to be common nomenclature...) $\endgroup$ Oct 20 '17 at 18:26
4
$\begingroup$

I'm somewhat confused by your points -- normalization is an invertible procedure so in terms of fitting one loses nothing by doing it, and it can often by very helpful when regressors are of different scales. The issue with nonstationarity only comes into play if your test set has a different distribution, in which case attempting to normalize by the estimated parameters of the training data will not help. For the training set, it is given to you already, so unless you are thinking of some sort of online algorithm you can always determine the mean and feature-wise variance.

Boolean and categorical regressors lead to residual distributions that don't follow the assumptions that lead to the clean solutions of standard linear regression, which is why they don't work well, is what I'd say. That and/or you implicitly want to constrain the space of your regression vector.

$\endgroup$
2
  • $\begingroup$ true, and your normalization parameters won't be useful. Can you explain what your objection based on nonstationarity is again? $\endgroup$ Oct 4 '17 at 7:57
  • $\begingroup$ It's a crappy estimator of the distributional mean, sure, but it still fulfills the function of putting your data close to zero (assuming the data distribution is stationary), which is useful for many algorithms. Plus I don't understand what you mean by non-stationary regressors still. If they're not stationary in any sense then how do you generalize from training to test time at all? $\endgroup$ Oct 4 '17 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.