16
$\begingroup$

Under what circumstances should the data be normalized/standardized when building a regression model. When i asked this question to a stats major, he gave me an ambiguous answer "depends on the data".

But what does that really mean? It should either be an universal rule or a check list of sorts where if certain conditions are met then the data either should/ shouldn't be normalized.

$\endgroup$
  • 1
    $\begingroup$ It should either be an universal rule or a check list of sorts where if certain conditions are met then the data either should/ shouldn't be normalized. Can you justify that? $\endgroup$ – Matthew Drury Mar 15 '16 at 21:45
  • 1
    $\begingroup$ Are you asking about standard linear regression, or about penalized methods like ridge regression or LASSO? $\endgroup$ – EdM Mar 15 '16 at 21:46
  • $\begingroup$ @MatthewDrury: What i mean is either data should be normalized for building all regression models (OLS, Logistic etc) or it should be done when so and so conditions are not satisfied like non-constant variance..etc (hypothetically speaking) $\endgroup$ – Raj Mar 15 '16 at 21:50
  • 1
    $\begingroup$ This page has an extensive discussion of situations in which normalization/standardization might be important. Please look it over and edit your question to include any issues that you believe weren't discussed adequately there. $\endgroup$ – EdM Mar 15 '16 at 21:59
  • 1
    $\begingroup$ Much software will automatically standardize the variables for its internal calculations. Could you clarify, then, whether you are asking about how to compute numerical solutions or about whether it is actually necessary to standardize the variables before applying regression software? $\endgroup$ – whuber Mar 15 '16 at 22:47
10
$\begingroup$

Sometimes standardization helps for numerical issues (not so much these days with modern numerical linear algebra routines) or for interpretation, as mentioned in the other answer. Here is one "rule" that I will use for answering the answer myself: Is the regression method you are using invariant, in that the substantive answer does not change with standardization? Ordinary least squares is invariant, while methods such as lasso or ridge regression are not. So, for invariant methods there is no real need for standardization, while for non-invariant methods you should probably standardize. (Or at least think it through).

The following is somewhat related: Dropping one of the columns when using one-hot encoding

$\endgroup$
5
$\begingroup$

It sometimes makes interpretation easier if you subtract the mean or some number within the range of the actual values as this can make the intercept more meaningful. For instance if you have people aged 65 and over subtract 65 and then the intercept is the predicted value for a 65-year old rather than a neonate. If you have non-linear terms like powers this makes them less correlated and so you can see more easily what is going on. It also may make life easier to scale the predictor so as to move the coefficients into a more printable range. For instance converting days into weeks or months. Other than that it should not matter. I suppose some of what I have just written may be what your friend meant by it depends on the data.

$\endgroup$
  • 2
    $\begingroup$ The correlation between predictors is unaffected by subtracting means, so nothing is less correlated than before. $\endgroup$ – Nick Cox Mar 16 '16 at 8:17
  • $\begingroup$ You are right @Nick, apologies. It is only non-linear terms for which it helps. $\endgroup$ – mdewey Mar 16 '16 at 11:32
  • $\begingroup$ "non-linear terms like powers this makes them less correlated" this is a general truth because to me it does sounds fine the connection between correlation and the non linearity. Would not a variable in a power be fully correlated with a response Y? I ask this because after this infer that the standardization in a ridged regression linear model sounds unnecessary. Thank you $\endgroup$ – nikolaos_mparoutis May 10 at 14:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.