# Upper bound for variance of $\hat{\beta}$ in multiple linear regression

The variance of the beta estimator in an ordinary-least-squares multiple linear regression to express $$Y$$ as a (linear) function of $$X$$, $$\hat{\beta}$$, can be expressed as (knowing $$X$$ and $$\sigma^2$$ the variance of the residuals, provided the Gauss-Markov assumptions hold):

$$Var[\hat{\beta}|X] = \sigma^2 (X^TX)^{-1}$$

Starting from this point, is there a known (and as tight as possible) upper bound (in $$\mathbb{R}$$) for all $$Var[\hat{\beta_j}|X] = \sigma^2 (X^TX)_{jj}^{-1}$$ with $$j > 0$$ (the constant coefficient of the linear regression being out of the scope)?

For sure the upper bound will depend on the regression data, like $$\sigma^2$$, $$\sigma_X^2$$, $$\sigma_Y^2$$. But this upper bound would be in $$\mathbb{R}$$, while the matrix form can be complicated to comprehend in the most general case.

If such an upper bound is not known, does there exist an expression for each $$\hat{\beta_j}$$ that is not in matrix form, and from which an upper bound can be more easily derived?

• You can make an arbitrarily stupid estimator of $\beta$. You can even make it "unbiased" by taking your BLUE estimator and literally adding random noise to it. Mar 30, 2021 at 19:40
• First, don't you mean $\sigma^2(X^\prime X)^{-1}_{jj}$? Second, given that scaling all entries in $X$ by any nonzero value $\lambda$ multiplies this expression by $\lambda^{-2},$ there clearly are no universal ($X$-independent) bounds.
– whuber
Mar 30, 2021 at 19:40
• @StatX Are you sure you mean upper bound? Because technically the $Var(\hat{\beta})$ could reach infinity if $\sigma^{2}$ is large and/or the moment matrix $(X^\mathsf{T}X)$ is nearly singular. Mar 30, 2021 at 19:41
• Sorry, I was unclear. I am aware that the upper bound will depend on $\sigma^2$, $\sigma_X^2$, etc. I am only wondering if there would be an upper bound to get "rid of" the matrix form. I will modify the question. Mar 30, 2021 at 19:44

A conceptually simple version of the variance is $$\mathrm{var}[\hat\beta]_{jj}=\frac{\mathrm{var}[Y|X]}{n\mathrm{var}[X_j|\textrm{other Xs}]}$$
Even fixing the distribution of each $$X$$ separately and of $$Y$$, the denominator can be an arbitrarily small positive number or zero, so the variance can be an arbitrarily large finite number or infinite.
Bounding the individual covariance terms is also not sufficient to get you anywhere: when the number of variables $$p$$ is large it is possible to get a zero denominator with all the covariances being small (about $$-1/p$$).
Since $$\mathrm{var}[X_j|\textrm{other Xs}]\leq \mathrm{var}[X_j]$$, a lower bound on the variance of $$\beta_j$$ is available from the separate distributions of $$Y$$ and the $$X$$s.
• For the simple linear regression I find a slightly different formula, where the term on the right is multiplied by $(1-\rho^2)$ where $\rho$ is the correlation between $X$ and $Y$. Mar 31, 2021 at 10:51
• Yes, thanks; corrected. It's the residual variance in $Y$, not the marginal variance. Mar 31, 2021 at 20:55