31
$\begingroup$

Under what circumstances should the data be normalized/standardized when building a regression model. When i asked this question to a stats major, he gave me an ambiguous answer "depends on the data".

But what does that really mean? It should either be an universal rule or a check list of sorts where if certain conditions are met then the data either should/ shouldn't be normalized.

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ It should either be an universal rule or a check list of sorts where if certain conditions are met then the data either should/ shouldn't be normalized. Can you justify that? $\endgroup$
    – Matthew Drury
    Mar 15, 2016 at 21:45
  • 1
    $\begingroup$ Are you asking about standard linear regression, or about penalized methods like ridge regression or LASSO? $\endgroup$
    – EdM
    Mar 15, 2016 at 21:46
  • $\begingroup$ @MatthewDrury: What i mean is either data should be normalized for building all regression models (OLS, Logistic etc) or it should be done when so and so conditions are not satisfied like non-constant variance..etc (hypothetically speaking) $\endgroup$
    – Raj
    Mar 15, 2016 at 21:50
  • 1
    $\begingroup$ This page has an extensive discussion of situations in which normalization/standardization might be important. Please look it over and edit your question to include any issues that you believe weren't discussed adequately there. $\endgroup$
    – EdM
    Mar 15, 2016 at 21:59
  • 1
    $\begingroup$ Much software will automatically standardize the variables for its internal calculations. Could you clarify, then, whether you are asking about how to compute numerical solutions or about whether it is actually necessary to standardize the variables before applying regression software? $\endgroup$
    – whuber ♦
    Mar 15, 2016 at 22:47

2 Answers 2

Reset to default
24
$\begingroup$

Sometimes standardization helps for numerical issues (not so much these days with modern numerical linear algebra routines) or for interpretation, as mentioned in the other answer. Here is one "rule" that I will use for answering the answer myself: Is the regression method you are using invariant, in that the substantive answer does not change with standardization? Ordinary least squares is invariant, while methods such as lasso or ridge regression are not. So, for invariant methods there is no real need for standardization, while for non-invariant methods you should probably standardize. (Or at least think it through).

The following is somewhat related: Dropping one of the columns when using one-hot encoding

Improve this answer
$\endgroup$
1
  • $\begingroup$ by OLS being "invariant", you mean that it is "scale invariant", where scale is the range of the numbers in the predictors/columns fed to OLS? $\endgroup$
    – Naveen Reddy Marthala
    Mar 30, 2023 at 10:39
7
$\begingroup$

It sometimes makes interpretation easier if you subtract the mean or some number within the range of the actual values as this can make the intercept more meaningful. For instance if you have people aged 65 and over subtract 65 and then the intercept is the predicted value for a 65-year old rather than a neonate. If you have non-linear terms like powers this makes them less correlated and so you can see more easily what is going on. It also may make life easier to scale the predictor so as to move the coefficients into a more printable range. For instance converting days into weeks or months. Other than that it should not matter. I suppose some of what I have just written may be what your friend meant by it depends on the data.

Improve this answer
$\endgroup$
4
  • 4
    $\begingroup$ The correlation between predictors is unaffected by subtracting means, so nothing is less correlated than before. $\endgroup$
    – Nick Cox
    Mar 16, 2016 at 8:17
  • $\begingroup$ You are right @Nick, apologies. It is only non-linear terms for which it helps. $\endgroup$
    – mdewey
    Mar 16, 2016 at 11:32
  • $\begingroup$ "non-linear terms like powers this makes them less correlated" this is a general truth because to me it does sounds fine the connection between correlation and the non linearity. Would not a variable in a power be fully correlated with a response Y? I ask this because after this infer that the standardization in a ridged regression linear model sounds unnecessary. Thank you $\endgroup$
    – nikolaosmparoutis
    May 10, 2019 at 14:58
  • $\begingroup$ @NickCox: How about when dividing in case we want to standardize. Don't we get unitless coefficients? $\endgroup$
    – MSIS
    Dec 19, 2019 at 0:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.