# distribution of least squares estimates for random design regression

Suppose we consider the least squares problem where the objective is to find $$\beta$$ that minimizes $$(\boldsymbol{y} - \boldsymbol{X} \beta)^T (\boldsymbol{y} - \boldsymbol{X} \beta)$$.

We know that an estimator for $$\beta$$ is given by $$\hat{\beta}= (\boldsymbol{X}^T \boldsymbol{X})^{-1} \boldsymbol{X}^T \boldsymbol{y}$$.

Under fixed design regression, I am aware that if $$\boldsymbol{y}$$ is corrupted by additive Gaussian noise with mean 0, then $$\hat{\beta}$$ is an unbiased estimator and has a Gaussian distribution.

Under random design regression in which $$\boldsymbol{X}$$ is also random, are there any results on the statistics of $$\hat{\beta}$$ (say expected value of $$\hat{\beta}$$) as well as its distribution assuming that $$\boldsymbol{y}$$ is still corrupted by 0 mean Gaussian noise and that $$\boldsymbol{X}$$ has a specified distribution?

Are there any references on this? Thanks!

• It's the same, because in both situations $\hat\beta$ estimates parameters of the conditional response of $y.$ – whuber Sep 20 '20 at 22:44
• Thanks for your reply @whuber. Do you care to elaborate? Conditional on X, I can see what you mean. But what about the unconditional case? – secondrate Sep 21 '20 at 6:55
• There is no "unconditional case:" that's not a regression problem. – whuber Sep 21 '20 at 13:01
• The properties that you like to be true are true unconditionally because they are true conditionally. For example, under the usual assumptions, the 95% confidence interval for $\beta_1$ has coverage level 95%, conditional on the $X$ data. The unconditional coverage level is the average coverage, averaged over all $X$. But for each realization of $X$, it is 95%, and the average of 95%, 95%, 95%, 95%..... is still 95%. – BigBendRegion Sep 21 '20 at 14:06
• Hm, can you show that $E[\hat{\beta}]$ is unbiased in the random design case? How would the calculation proceed when it's hard to compute, for example, $E[(X^TX)^{-1}X^T]$? – secondrate Sep 24 '20 at 8:58