Suppose we consider the least squares problem where the objective is to find $\beta$ that minimizes $(\boldsymbol{y} - \boldsymbol{X} \beta)^T (\boldsymbol{y} - \boldsymbol{X} \beta)$.

We know that an estimator for $\beta$ is given by $\hat{\beta}= (\boldsymbol{X}^T \boldsymbol{X})^{-1} \boldsymbol{X}^T \boldsymbol{y}$.

Under fixed design regression, I am aware that if $\boldsymbol{y}$ is corrupted by additive Gaussian noise with mean 0, then $\hat{\beta}$ is an unbiased estimator and has a Gaussian distribution.

Under random design regression in which $\boldsymbol{X}$ is also random, are there any results on the statistics of $\hat{\beta}$ (say expected value of $\hat{\beta}$) as well as its distribution assuming that $\boldsymbol{y}$ is still corrupted by 0 mean Gaussian noise and that $\boldsymbol{X}$ has a specified distribution?

Are there any references on this? Thanks!

  • $\begingroup$ It's the same, because in both situations $\hat\beta$ estimates parameters of the conditional response of $y.$ $\endgroup$ – whuber Sep 20 '20 at 22:44
  • $\begingroup$ Thanks for your reply @whuber. Do you care to elaborate? Conditional on X, I can see what you mean. But what about the unconditional case? $\endgroup$ – secondrate Sep 21 '20 at 6:55
  • $\begingroup$ There is no "unconditional case:" that's not a regression problem. $\endgroup$ – whuber Sep 21 '20 at 13:01
  • $\begingroup$ The properties that you like to be true are true unconditionally because they are true conditionally. For example, under the usual assumptions, the 95% confidence interval for $\beta_1$ has coverage level 95%, conditional on the $X$ data. The unconditional coverage level is the average coverage, averaged over all $X$. But for each realization of $X$, it is 95%, and the average of 95%, 95%, 95%, 95%..... is still 95%. $\endgroup$ – BigBendRegion Sep 21 '20 at 14:06
  • $\begingroup$ Hm, can you show that $E[\hat{\beta}]$ is unbiased in the random design case? How would the calculation proceed when it's hard to compute, for example, $E[(X^TX)^{-1}X^T]$? $\endgroup$ – secondrate Sep 24 '20 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.