Consider the linear regression model with one non-stochastic predictor: $Y = x \beta + \varepsilon$, where $Y \in \mathbb R^n$, $x \in \mathbb R^n$, $\beta \in \mathbb R$, and $\varepsilon \sim \mathrm{N}_n(0, \Sigma)$ for some known, positive definite $\Sigma$. Let $y$ denote the observed response. The maximum likelihood estimate of $\beta$ is $$\hat{\beta} = \operatorname{argmin}_{b \in \mathbb R} (y - x b)^T\Sigma^{-1}(y - xb) = y^T\Sigma^{-1}x / x^T\Sigma^{-1}x.$$

Question: Assuming $x^Tx > 0$, is $\vert \hat{\beta}\vert$ bounded as a function of $\Sigma$, for fixed $y$ and $x$?

Two special cases where the answer is 'yes'

(1) If we only consider i.i.d. errors so that $\Sigma = cI_n$, $c > 0$, then $c^{-1}$ cancels in the numerator and denominator so $\hat{\beta}$ does not depend on $\Sigma$ at all.

(2) Suppose $\Sigma^{-1} = D = \operatorname{diag}(d_1, \dots, d_n)$, then if $x_{-}$ denotes the smallest non-zero $\vert x_i\vert$ ($i = 1, \dots, n$),

$$ \hat{\beta} = \frac{\sum_{i = 1}^n d_i y_i x_i}{\sum_{i = 1}^n d_i x_i^2} \leq \Vert y\Vert_{\infty} \frac{\sum_{i = 1}^n d_i \vert x_i\vert}{\sum_{i = 1}^n d_i x_i^2} \leq \Vert y\Vert_{\infty}x_{-}^{-2} \frac{\sum_{i \in \mathcal I} d_i \vert x_i\vert}{\sum_{i \in \mathcal I} d_i} \leq \Vert y\Vert_\infty x_{-}^{-2} \Vert x\Vert_\infty, $$ where $\mathcal I$ is the (non-empty) set of all $i$ such that $x_i \neq 0$.


The proof for the diagonal case would work also in the general case after spectral decomposing $\Sigma^{-1} = U^T D U$ and replacing $y$ by $\tilde{y} = Uy$ and $x$ by $\tilde{x} = Ux$, if $\tilde{x}_-$ was lower bounded as a function of $U$. I don't see how it can be, however, and hence suspect the answer is 'no' in general; I have not been able to prove it.

  • 2
    $\begingroup$ You can choose $\Sigma^{-1}$ to have an infinitesimally small eigenvalue in the $x$ direction and finite eigenvalues in all orthogonal directions. Unless $y$ is parallel to $x,$ this will assure a very large quotient, demonstrating there is no bound unless you can restrict $\Sigma$ to keep this from happening. (1) is an extreme example of such a restriction and (2) is a more general example. $\endgroup$ – whuber Aug 15 '18 at 17:08
  • $\begingroup$ Thanks @whuber, If you write it as an answer I'll accept it. If not I'll come back later and write an answer based on your comment. $\endgroup$ – ekvall Aug 15 '18 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.