Correlation of t-statistics for two correlated predictors in two simple linear regressions with common y

Question

Say we run two simple linear regressions with common $y$ but different and correlated predictors $x_1, x_2$: $$y = \beta_{01} + \beta_1 x_1 + \epsilon_1,$$ $$y = \beta_{02} + \beta_2 x_2 + \epsilon_2.$$

We obtain the t-statistics $t_1 = \hat\beta_1/\hat{se}(\hat\beta_1), t_2 = \hat\beta_2/\hat{se}(\hat\beta_2)$. The t-statistics can be expressed as a function of sample correlation $r$, $t = \sqrt{(n-2)/(r^{-2}-1)}.$ My questions are:

1. Is $\rho(t_1, t_2)=\rho(x_1, x_2)$ approximately true? $\rho$ represents population correlation. A simple simulation shows that the two correlations are pretty close. Normality assumption can be assumed if needed.
2. If Q1 is untrue, would $\rho(t_1, t_2)$ be unrelated to $y$?

Any suggestions would be greatly appreciated.

Answer 1

I think Q1 is false and a "simple" simulation can confirm (see below). However, there is probably an equation that can relate $\rho(t_1, t_2)$ to $\rho(x_1, x_2)$. For Q2, I think, it is basically related to the residual error in $y$.

The way I see it is basically to take the "complete" linear model: $y=\beta_1x_1+\beta_2x_2+\epsilon _y$,or in matrix algebra : $y=BX + \epsilon_y$. Here, the $\beta$ in $B$ are partial regression coefficients, thus they are not the same as in the OP (where they are simple regression coefficients).

Let's define $\Sigma$, the covariance among $X$, then we know: $B$, $\text{var}(B)$ and $(1-R^2)$, which is the residual error variance in $y$.

Regarding OP, we can already compute $[\beta_1^{*},\beta_2^{*}] =B^{*}= B\Sigma$. Since we are regarding the linear model as two different models, there is then two $R^2$ values, one for each linear model, which are $R^{2*}=(B\Sigma)^2$.

We now see that $\Sigma$ becomes $\Sigma^*=\text{diag}(\Sigma)$, which is a square matrix with variances in the diagonal and 0 in the off-diagonal elements.

On the $t$ values, in the complete linear model we get :

$t=\frac{B}{(\frac{(1-R^2)}{ \text{diag}(\Sigma^{-1})})}$ and $t^*=\frac{B^*}{(\frac{(1-R^{2*})}{ \text{diag}(\Sigma^{*-1})})}$. So the inverse of $\Sigma$ and $\Sigma^*$, the regression coefficients, and the $R^2$ are very different. We see where the similarities come up, but know the complete relation.

For the simulation :

n = 10000
b1 = .4
rho = .3
b2 = .2 
reps = 1000000
tvalue = matrix(0,4,reps)
for(i in 1:reps){
  X = MASS::mvrnorm(n=n,mu=mu,Sigma=S)  
  Y = t(B%*%t(X) + rnorm(n) * (sqrt(1-R2)))
  res0 = lm(Y~X[,1]+X[,2])
  res1 = lm(Y~X[,1])
  res2 = lm(Y~X[,2])
  tvalue[,i] = c(summary(res0)$coefficients[2:3,3],
  summary(res1)$coefficients[2,3],
  summary(res2)$coefficients[2,3])
  
}
> round(cov(t(tvalue)),3)
       [,1]   [,2]   [,3]   [,4]
[1,]  1.198 -0.242  1.155 -0.068
[2,] -0.242  1.020 -0.062  0.917
[3,]  1.155 -0.062  1.273  0.268
[4,] -0.068  0.917  0.268  1.080
> round(cor(t(tvalue)),3)
       [,1]   [,2]   [,3]   [,4]
[1,]  1.000 -0.219  0.935 -0.060
[2,] -0.219  1.000 -0.055  0.874
[3,]  0.935 -0.055  1.000  0.229
[4,] -0.060  0.874  0.229  1.000

We see that they are close (.229 is close to .3), but not the same.