Say we run two simple linear regressions with common $y$ but different and correlated predictors $x_1, x_2$: $$y = \beta_{01} + \beta_1 x_1 + \epsilon_1,$$ $$y = \beta_{02} + \beta_2 x_2 + \epsilon_2.$$

We obtain the t-statistics $t_1 = \hat\beta_1/\hat{se}(\hat\beta_1), t_2 = \hat\beta_2/\hat{se}(\hat\beta_2)$. The t-statistics can be expressed as a function of sample correlation $r$, $t = \sqrt{(n-2)/(r^{-2}-1)}.$ My questions are:

  1. Is $\rho(t_1, t_2)=\rho(x_1, x_2)$ approximately true? $\rho$ represents population correlation. A simple simulation shows that the two correlations are pretty close. Normality assumption can be assumed if needed.
  2. If Q1 is untrue, would $\rho(t_1, t_2)$ be unrelated to $y$?

Any suggestions would be greatly appreciated.

  • $\begingroup$ Please clarify what you mean by "correlated predictors:" would these be (a) random variables $(x_1,x_2)$ or (b) fixed values that, as a dataset, have nonzero correlation? $\endgroup$ – whuber Feb 3 at 18:57
  • $\begingroup$ Thanks, @whuber! This is an interesting question. I know we usually assume predictors are fixed in regression. I am not particularly sure which one makes more sense, but I guess b) would make the problem simpler? In terms of science, we know $x_1$ and $x_2$ are dependent, and this dependence can be reflected via their correlation in the measured data. $\endgroup$ – Randel Feb 3 at 22:06
  • $\begingroup$ It sounds, then, like you intend the former interpretation (a). That would be the point of view adopted, say, in an experiment whose output is random values $(x_1,x_2,y)$ and you wish to regress $y$ on the $x_i.$ I have a second question: your speculation in (1) appears to hold provided $(x_1,x_2,y)$ have an approximate multivariate Normal distribution. But without that distributional assumption, the correlation between the $t_i$ can be (literally) anything between $-1$ and $1$ regardless of the correlation between the $x_i.$ What distributional assumptions are you making? $\endgroup$ – whuber Feb 3 at 23:01
  • $\begingroup$ @whuber In reality, $x$ is discrete with 0,1,2 values, which may be assumed to follow a binomial distribution. But it is promising if 1) holds given multivariate normal assumption. Would you mind sharing your thoughts on how 1) is true under normality and untrue under other distributions? Many thanks! $\endgroup$ – Randel Feb 3 at 23:23
  • 1
    $\begingroup$ I am really interested to know the answer to this question. Not sure where to start to find the answer as there is a lot involved. $\endgroup$ – POC Feb 11 at 1:06

I think Q1 is false and a "simple" simulation can confirm (see below). However, there is probably an equation that can relate $\rho(t_1, t_2)$ to $\rho(x_1, x_2)$. For Q2, I think, it is basically related to the residual error in $y$.

The way I see it is basically to take the "complete" linear model: $y=\beta_1x_1+\beta_2x_2+\epsilon _y$,or in matrix algebra : $y=BX + \epsilon_y$. Here, the $\beta$ in $B$ are partial regression coefficients, thus they are not the same as in the OP (where they are simple regression coefficients).

Let's define $\Sigma$, the covariance among $X$, then we know: $B$, $\text{var}(B)$ and $(1-R^2)$, which is the residual error variance in $y$.

Regarding OP, we can already compute $[\beta_1^{*},\beta_2^{*}] =B^{*}= B\Sigma$. Since we are regarding the linear model as two different models, there is then two $R^2$ values, one for each linear model, which are $R^{2*}=(B\Sigma)^2$.

We now see that $\Sigma$ becomes $\Sigma^*=\text{diag}(\Sigma)$, which is a square matrix with variances in the diagonal and 0 in the off-diagonal elements.

On the $t$ values, in the complete linear model we get :

$t=\frac{B}{(\frac{(1-R^2)}{ \text{diag}(\Sigma^{-1})})}$ and $t^*=\frac{B^*}{(\frac{(1-R^{2*})}{ \text{diag}(\Sigma^{*-1})})}$. So the inverse of $\Sigma$ and $\Sigma^*$, the regression coefficients, and the $R^2$ are very different. We see where the similarities come up, but know the complete relation.

For the simulation :

n = 10000
b1 = .4
rho = .3
b2 = .2 
reps = 1000000
tvalue = matrix(0,4,reps)
for(i in 1:reps){
  X = MASS::mvrnorm(n=n,mu=mu,Sigma=S)  
  Y = t(B%*%t(X) + rnorm(n) * (sqrt(1-R2)))
  res0 = lm(Y~X[,1]+X[,2])
  res1 = lm(Y~X[,1])
  res2 = lm(Y~X[,2])
  tvalue[,i] = c(summary(res0)$coefficients[2:3,3],
> round(cov(t(tvalue)),3)
       [,1]   [,2]   [,3]   [,4]
[1,]  1.198 -0.242  1.155 -0.068
[2,] -0.242  1.020 -0.062  0.917
[3,]  1.155 -0.062  1.273  0.268
[4,] -0.068  0.917  0.268  1.080
> round(cor(t(tvalue)),3)
       [,1]   [,2]   [,3]   [,4]
[1,]  1.000 -0.219  0.935 -0.060
[2,] -0.219  1.000 -0.055  0.874
[3,]  0.935 -0.055  1.000  0.229
[4,] -0.060  0.874  0.229  1.000

We see that they are close (.229 is close to .3), but not the same.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.