1
$\begingroup$

I am currently trying to become familiar with design of experiments with the book “Optimal design of experiments” by Goos and Jones. In chapter 2, they discuss the use of a Plackett–Burman type design (6 factors, 12 experiments), and in chapter 2.3 (“Peek into the black box”), they introduce the math. I am currently struggling with chapter 2.3.4 (“Ordinary least squares estimates”), more specifically with some of the equations.

Quote:

The ordinary least squares estimator of the vector of unknown model >coefficients $\beta $ is $$\hat{\beta }=({X}'X)^{-1}{X}'Y.$$

Ok, I understand that, assuming X' is the transposed of X. Then they continue

The variance-covariance matrix of this estimator is $$var(\hat{\beta })=\sigma _{\varepsilon } ^{2}({X}'X)^{-1}.$$

Here, I am getting confused. I thought that $\beta$ (the "real" values) and $\hat{\beta}$ (the calculated values) are vectors, and that the elements $\beta _i$ are scalars. However, the equation and the wording suggest that $\hat{\beta}$ is a matrix, and that the elements $\hat{\beta}_i$ are vectors. Moreover, how is it possible to calculate anything related to $\beta$ without the regressand vector Y. What am I missing here?

Further down in the text, they write

Note that the variance-covariance matrix of the estimator is directly proportional to the error variance, which is unknown. We can estimate the error variance using the mean squared error: $$\hat{\sigma} > _{\varepsilon } ^{2} = \frac{1}{n-p}(Y- X \hat{\beta} )' (Y- X \hat{\beta} ).$$

I can understand that equation. At least, it makes sense that the difference between Y and $X\hat{\beta }$ (or $\hat{Y}$) is proportional to $\hat{\sigma} _{\varepsilon }$.

But I would really like to understand the other equations as well.

Regards, Soltub

Improve this question
$\endgroup$
1
$\begingroup$

I think your main point of confusion might just be a misunderstanding of notation -- you are indeed correct that $\hat{\beta}$ should be a vector. However, $Var(\hat{\beta})$ is a matrix called (as you have noted above) the variance-covariance (or just covariance) matrix. More specifically, for parameter vector $\hat{\beta} \in \mathbb{R}^n$, $Var(\hat{\beta}) \in \mathbb{R}^{n \times n}$.

Why is it a matrix instead of a vector?

Well, there's nothing really stopping you from defining some quantity of $\hat{\beta}$ that measures element-wise variance of each element of the vector, but it's a more informative measure of how a random vector varies, since variances in one element may correlate to variances in another element.

In particular, the OLS estimator has sampling distribution $\mathcal{N}(\beta, \sigma_\varepsilon^2 I)$ -- a multivariate Gaussian. At risk of some hand-waviness -- you can't parameterize a multivariate Gaussian with only the element-wise variances of the vector elements; you need to take into account the dependencies between elements of the vector. To explore this further, you can try playing with the covariance matrix of a 2-D multivariate Gaussian, and observe how the shape changes.

Derivation

There's a lot of possible derivations for the variance of the OLS estimator; here's one I like. I'll try to keep the same notation you used.

We have our usual setup: $$Y = X\beta + \varepsilon, \;\varepsilon\sim \mathcal{N}(0, \sigma_\varepsilon^2 I)$$

By definition: $$Var(\hat{\beta}) = \mathbb{E}[(\hat{\beta} - \beta)(\hat{\beta} - \beta)'].$$

Let's find $\hat{\beta} - \beta$ as an intermediate step. Recall that $$\hat{\beta} = (X'X)^{-1}X' Y,$$ so, substituting, we have $$\hat{\beta} = (X'X)^{-1}X' (X\beta + \varepsilon)$$ $$ = \beta + (X'X)^{-1}X'\varepsilon$$ so that $$\hat{\beta} - \beta = (X'X)^{-1}X'\varepsilon.$$ Then, substituting into the original expression for variance:

$$Var(\hat{\beta}) = \mathbb{E}[(X'X)^{-1}X'\varepsilon\varepsilon'X(X'X)^{-1}]$$ $$=(X'X)^{-1}X'\mathbb{E}[\varepsilon\varepsilon']X(X'X)^{-1}$$.

Note that the $\mathbb{E}[\varepsilon\varepsilon']$ term is simply the covariance of random vector $\varepsilon$, which is given to us as $\sigma_\varepsilon^2 I$. We can substitute, multiply out the $I$, and rearrange constants:

$$=(X'X)^{-1}X'\sigma_\varepsilon^2 I X(X'X)^{-1}$$ $$=\sigma_\varepsilon^2 \cdot (X'X)^{-1}X'X(X'X)^{-1}$$ $$=\sigma_\varepsilon^2 (X'X)^{-1}$$

as needed. $\blacksquare$

I skipped a step where I sneakily removed the transpose from $(X'X)^{-1}$; since $X'X$ is symmetric, $(X'X)^{-1}$ is as well, which is proven here.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you very much or your answer. Though I will need some time to fully appreciate it, you have clarified two points: 1. the elements in $\hat{\beta }$ are not numbers/calculatable scalars, but distributions and 2. I need to get a better understanding of what a variance-covariance matrix actually is. I have seen (don’t remember where) how to calculate the variance-covariance matrix of two equal length vectors of scalars, but that is obviously not the complete picture. $\endgroup$ – Soltub Jan 4 at 19:31
  • $\begingroup$ Well, in practice, you will calculate some fixed value for $\hat{\beta}$; the variance of an estimator is there to help us quantify how close our estimator will be to the true value $\beta$ (since the OLS estimator is unbiased) on a finite sample. You can think of $\hat{\beta}$ in a couple of ways: 1) as a random vector, which is useful when you want to reason about the variance of the estimator, or 2) as the solution to your least-squares objective given data $X$, which can also be viewed as a "sample" from the Gaussian derived above. $\endgroup$ – tchainzzz Jan 5 at 1:44
  • $\begingroup$ To calculate the variance-covariance matrix of a random vector $x$, you simply take $\mathbb{E}[(x - \mu_x)(x - \mu_x)^\top]$. This is what we did above, as $\mathbb{E}[\hat{\beta}] = \beta$ (OLS is unbiased) -- you can verify this; I imagine your book has a proof. You can think of it as a generalization of scalar variance (i.e. pretend $x \in \mathbb{R}$, and you get the definition of variance) -- it's a function of a random variable, whether scalar, vector, or otherwise. $\endgroup$ – tchainzzz Jan 5 at 1:47
  • $\begingroup$ Thanks for your additional comments. Some further steps of my way to enlightenment. BTW, my book is an interesting read, but the mathematical treatment is not optimal for me. On the one hand, they elaborate on things that I would consider obvious (e. g. that you need to add a column with 1's to your design matrix if you want your model to have an intercept, or the conversion from design to engineering coordinates), but for other topics, they are moving too fast. I will keep reading and meditating, and probably get an introductory book. $\endgroup$ – Soltub Jan 5 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.