0
$\begingroup$

I am currently reading Regression Analysis by Example by Chatterjee and Hadi and they state "If we are fitting an intercept model as in (3.16), we need to center and scale the variables" (pg.65). I understand that standardizing is efficient when variables are on different scales and allows us to assess the standardized coefficients in terms of the marginal effects of the predictor variable in standard deviation units. However, I am unclear as to if standardizing is truly necessary if we back-transform to identify the original coefficients.

From several posts I have read it seems that standardizing doesn't play a significant role in linear regression. Rather, it appears to be a precautionary measure done by most statisticians. Is this accurate? If I have a design matrix containing several variables on different scales should I be standardizing the data before performing OLS even if I back-transform after performing regression (as described on pg.67 of the book)?

$\endgroup$
5
  • 7
    $\begingroup$ As a statistician I very rarely standardise data and I don't know any other statisticians that do. $\endgroup$ – Robert Long Mar 15 at 13:13
  • $\begingroup$ Technically, for OLS, apart from the interpretation of the relative coefficient size, standardisation and centering don't make a difference, due to the equivariance properties of OLS. This is different for some other methods such as Lasso. $\endgroup$ – Lewian Mar 15 at 13:14
  • $\begingroup$ @RobertLong Very rarely? I can understand linear models, but there are many models which work better with standardized or normalized variables, for ex. everything having to do with distances. $\endgroup$ – user2974951 Mar 15 at 13:50
  • 3
    $\begingroup$ @user2974951 the question is about linear regression, so that's what I was referring to. $\endgroup$ – Robert Long Mar 15 at 15:40
  • $\begingroup$ This is a FAQ. A lot of similar posts $\endgroup$ – kjetil b halvorsen Mar 15 at 16:31
1
$\begingroup$

Standardization is optional for most cases of OLS regression. The story is a little different for multilevel models where centering really can impact the interpretability of your variables.

As you noted, there are certain benefits of standardized variables in OLS from a results interpretation perspective. Where I'm most familiar with the recommendation to standardize variables is when there is an interaction variable or when the predictors don't have a meaningful zero value. Since the excerpt mentions the intercept in particular, it may be that the recommendation for standardization is related to this latter point. The intercept of a regression will be the predicted y-value when x is equal to zero (the general idea is the same for multiple regression, but obviously there are more x-variables to consider). That intercept is fine regardless of whether variables are left untransformed, centered, or standardized. The transformation of the variables just helps shift the interpretation of the intercept (and coefficients) so that it is easier to look at the estimates and get a sense of what the equation is telling you. If the predictor is standardized, then the intercept becomes the expected y-value at the mean of the x variable. The same is true when the predictor is centered, but the advantage then of standardizing (dividing the mean-centered transformation by the standard deviation) is that the coefficient of the predictor is now in terms of standard deviation changes in the variable rather than unit changes.

Where you might start becoming concerned with the scale of the variables you're using in the regression is is if you're estimating multilevel models, expect that readers/users of your equation will interpret the magnitude of the coefficient as it's "importance", or if you're using Bayesian estimation (simplifying the scales of the variables usually makes for an easier-to-sample posterior space and makes prior specification a bit more streamlined). Other than that, and maybe a few other use cases, centering or scaling your variables is just to facilitate the interpretability of your final regression equation.

$\endgroup$
0
$\begingroup$

I am assuming if a model is "an intercept model" doesn't have anything with the conclusions. My belief is that working with standardized data provides some mathematical apparatus especially when they start to generalize things for multiple linear regression.

The other reason may be to make the regression coefficients unitless, by centering and scaling the variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.