The variance of the beta estimator in an ordinary-least-squares multiple linear regression to express $Y$ as a (linear) function of $X$, $\hat{\beta}$, can be expressed as (knowing $X$ and $\sigma^2$ the variance of the residuals, provided the Gauss-Markov assumptions hold):

$Var[\hat{\beta}|X] = \sigma^2 (X^TX)^{-1}$

Starting from this point, is there a known (and as tight as possible) upper bound (in $\mathbb{R}$) for all $Var[\hat{\beta_j}|X] = \sigma^2 (X^TX)_{jj}^{-1}$ with $j > 0$ (the constant coefficient of the linear regression being out of the scope)?

For sure the upper bound will depend on the regression data, like $\sigma^2$, $\sigma_X^2$, $\sigma_Y^2$. But this upper bound would be in $\mathbb{R}$, while the matrix form can be complicated to comprehend in the most general case.

If such an upper bound is not known, does there exist an expression for each $\hat{\beta_j}$ that is not in matrix form, and from which an upper bound can be more easily derived?

  • $\begingroup$ You can make an arbitrarily stupid estimator of $\beta$. You can even make it "unbiased" by taking your BLUE estimator and literally adding random noise to it. $\endgroup$ – AdamO Mar 30 at 19:40
  • $\begingroup$ First, don't you mean $\sigma^2(X^\prime X)^{-1}_{jj}$? Second, given that scaling all entries in $X$ by any nonzero value $\lambda$ multiplies this expression by $\lambda^{-2},$ there clearly are no universal ($X$-independent) bounds. $\endgroup$ – whuber Mar 30 at 19:40
  • $\begingroup$ @StatX Are you sure you mean upper bound? Because technically the $Var(\hat{\beta})$ could reach infinity if $\sigma^{2}$ is large and/or the moment matrix $(X^\mathsf{T}X)$ is nearly singular. $\endgroup$ – Durden Mar 30 at 19:41
  • $\begingroup$ Sorry, I was unclear. I am aware that the upper bound will depend on $\sigma^2$, $\sigma_X^2$, etc. I am only wondering if there would be an upper bound to get "rid of" the matrix form. I will modify the question. $\endgroup$ – StatX Mar 30 at 19:44

There is no non-trivial upper bound that doesn't depend on a lot of detail about the matrix.

A conceptually simple version of the variance is $$\mathrm{var}[\hat\beta]_{jj}=\frac{\mathrm{var}[Y|X]}{n\mathrm{var}[X_j|\textrm{other Xs}]}$$

Even fixing the distribution of each $X$ separately and of $Y$, the denominator can be an arbitrarily small positive number or zero, so the variance can be an arbitrarily large finite number or infinite.

Bounding the individual covariance terms is also not sufficient to get you anywhere: when the number of variables $p$ is large it is possible to get a zero denominator with all the covariances being small (about $-1/p$).

Since $\mathrm{var}[X_j|\textrm{other Xs}]\leq \mathrm{var}[X_j]$, a lower bound on the variance of $\beta_j$ is available from the separate distributions of $Y$ and the $X$s.

  • $\begingroup$ For the simple linear regression I find a slightly different formula, where the term on the right is multiplied by $(1-\rho^2)$ where $\rho$ is the correlation between $X$ and $Y$. $\endgroup$ – StatX Mar 31 at 10:51
  • $\begingroup$ Yes, thanks; corrected. It's the residual variance in $Y$, not the marginal variance. $\endgroup$ – Thomas Lumley Mar 31 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.