9
$\begingroup$

Collinearity can pose certain problems in various kinds of regression problem. In particular, it can make the parameter estimates have high variance and be unstable.

Various methods have been proposed to deal with this including ridge regression, partial least squares regression, principal components regression, dropping variables and getting more data.

One method that is controversial is standardizing or scaling the independent variables, with various experts saying it is a good (e.g. Garcia) or bad (e.g. Belsley) idea. Belsley's issue seems to be (in non-technical terms) that changing the IVs just pushes the problem under the rug. But other experts seem not to agree. And the authors tend to get rather heated in defending their own positions.

Back when I did my dissertation (on collinearity diagnostics) I found Belsley's arguments convincing, but that was long ago (I got my degree in 1999).

I'm looking either for expert guidance or for any current review article that is unbiased.

$\endgroup$
  • 3
    $\begingroup$ I have no modern references for you -- my go-to authority still is Belsley Kuh & Welsch 1981 -- but I can say that recent experience in fixing up certain regression software convinced me there is indeed some value in a preliminary standardization. In the application, one variable was time which, in this R framework, is represented in seconds since the beginning of 1970. As such it tended to be nine orders of magnitude greater than all the covariates. Simply standardizing time solved severe floating point problems occurring in the likelihood optimizer. $\endgroup$ – whuber Sep 5 at 15:07
  • 6
    $\begingroup$ Conceptually (not numerically), I still think Arthur Goldberger was spot on: "Econometrics texts devote many pages to the problem of multicollinearity in multiple regression, but they say little about the closely analogous problem of small sample size in estimating a univariate mean. Perhaps that imbalance is attributable to the lack of an exotic polysyllabic name for ‘small sample size.’ If so, we can remove that impediment by introducing the term micronumerosity" $\endgroup$ – CloseToC Sep 5 at 15:17
  • 1
    $\begingroup$ @Peter Flom: Consisteny with Whuber's comment, I ( very ) vaguely recall that standardizing by even just making the predictors have zero mean helped a lot. $\endgroup$ – mlofton Sep 5 at 15:22
2
+100
$\begingroup$

It was not so clear to me what sort of standardization was meant, and while looking for the history I picked up two interesting references.

This recent article has a historic overview in the introduction:

García, J., Salmerón, R., García, C., & López Martín, M. D. M. (2016). Standardization of variables and collinearity diagnostic in ridge regression. International Statistical Review, 84(2), 245-266

I found another interesting article that sort of claims to show that standardization, or centering, has no effect at all.

Echambadi, R., & Hess, J. D. (2007). Mean-centering does not alleviate collinearity problems in moderated multiple regression models. Marketing Science, 26(3), 438-445.


To me this criticism all seems a bit like missing the point about the idea of centering.

The only thing that Echambadi and Hess show is that the models are equivalent and that you can express the coefficients of the centered model in terms of the coefficients of the non-centered model, and vice versa (resulting in similar variance/error of the coefficients).

Echambadi and Hess' result is a bit trivial and I believe that this (those relations and equivalence between the coefficients) is not claimed to be untrue by anybody. Nobody claimed that those relations between the coefficients are not true. And it is not the point of centering variables.

The point of the centering is that in models with linear and quadratic terms you can choose different coordinate scales such that you end up working in a frame that has no or less correlation between the variables. Say you wish to express the effect of time $t$ on some variable $Y$ and you wish to do this over some period expressed in terms of years AD say from 1998 to 2018. In that case, what the centering technique means to resolve is that

"If you express the accuracy of the coefficients for the linear and quadratic dependencies on time, then they will have more variance when you use time $t$ ranging from 1998 to 2018 instead of a centered time $t^\prime$ ranging from -10 to 10".

$$Y = a + bt + ct^2$$

versus

$$Y = a^\prime + b^\prime(t-T) + c^\prime(t-T)^2$$

Of course, these two models are equivalent and instead of centering you can get the exact same result (and hence the same error of the estimated coefficients) by computing the coefficients like

$$\begin{array}{} a &=& a^\prime - b^\prime T + c^\prime T^2 \\ b &=& b^\prime - 2 c^\prime T \\ c &=& c^\prime \end{array}$$

also when you do ANOVA or use expressions like $R^2$ then there will be no difference.

However, that is not at all the point of mean-centering. The point of mean-centering is that sometimes one wants to communicate the coefficients and their estimated variance/accuracy or confidence intervals, and for those cases it does matter how the model is expressed.

Example: a physicists wishes to express some experimental relation for some parameter X as a quadratic function of temperature.

  T   X
  298 1230
  308 1308
  318 1371
  328 1470
  338 1534
  348 1601
  358 1695
  368 1780
  378 1863
  388 1940
  398 2047

would it not be better to report the 95% intervals for coefficients like

                 2.5 %      97.5 %

(Intercept)      1602       1621
T-348               7.87       8.26
(T-348)^2           0.0029     0.0166

instead of

                  2.5 %     97.5 %

(Intercept)       -839       816
T                   -3.52      6.05
T^2                  0.0029    0.0166

In the latter case the coefficients will be expressed by seemingly large error margins (but telling nothing about the error in the model), and in addition the correlation between the distribution of the error won't be clear (in the first case the error in the coefficients will not be correlated).

If one claims, like Echambadi and Hess, that the two expressions are just equivalent and the centering does not matter, then we should (as a consequence using similar arguments) also claim that expressions for model coefficients (when there is no natural intercept and the choice is arbitrary) in terms of confidence intervals or standard error are never making sense.

In this question/answer an image is shown that also presents this idea how the 95% confidence intervals do not tell much about the certainty of the coefficients (at least not intuitively) when the the errors in the estimates of the coefficients are correlated.

image

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.