10
$\begingroup$

For least squares with one predictor:

$y = \beta x + \epsilon$

If $x$ and $y$ are standardised prior to fitting (i.e. $\sim N(0,1)$), then:

  • $\beta$ is the same as the Pearson correlation coefficient, $r$.
  • $\beta$ is the same in the reflected regression: $x = \beta y + \epsilon$

For generalised least squares (GLS), does the same apply? I.e. if I standardise my data, can I obtained correlation coefficients directly from the regression coefficients?

From experimenting with data, the reflected GLS leads to different $\beta$ coefficients and also I'm not sure that I'm believing that the regression coefficients fit with my expected values for correlation. I know people quote GLS correlation coefficients, so I am wondering how they arrive at them and hence what they really mean?

$\endgroup$
4
$\begingroup$

The answer is yes, the linear regression coefficients are the correlations of the predictors with the response, but only if you use the correct coordinate system.

To see what I mean, recall that if $x_1, x_2, \ldots, x_n$ and $y$ are centered and standardized, then the correlation between each $x_i$ and $y$ is just the dot product $x_i^t y$. Also, the least squares solution to linear regression is

$$ \beta = (X^t X)^{-1} X^t y $$

If it so happens that $X^{t} X = I$ (the identity matrix) then

$$ \beta = X^t y $$

and we recover the correlation vector. It is often attractive to recast a regression problem in terms of predictors $\tilde{x}_i$ that satisfy $\tilde{X}^t \tilde{X} = I$ by finding appropriate linear combinations of the original predictors that make this relation true (or equivalently, a linear change of coordinates); these new predictors are called the principal components.

So overall, the answer to your question is yes, but only when the predictors are themselves uncorrelated. Otherwise, the expression

$$X^t X \beta = X^t y$$

shows that the betas must be mixed together with the correllations between the predictors themselves to recover the predictor-response correlations.

As a side note, this also explains why the result is always true for one variable linear regression. Once the predictor vector $x$ is standardized, then:

$$ x_0^t x = \sum_i x_{i} = 0 $$

where $x_0$ is the intercept vector of all ones. So the (two column) data matrix $X$ automatically satisfies $X^t X = I$, and the result follows.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.