# Relationship between sample size and parameter covariance matrix in OLS

I am dealing with a linear system of equations that I am solving by OLS:

$$\mathbf{y} = \mathbf{X} \mathbf{p} + \mathbf{e}$$

Where I have $$n$$ samples and $$k$$ parameters ($$\mathbf{X}$$ is an $$n \times k$$ matrix)

I would like to work out the relationship between samples size ($$n$$) and the parameter uncertainties contained within their covariance matrix ($$\mathbf{C_p}$$).

I have established numerically (by simulating an OLS with different $$n$$) that the parameter variance decreases ~exponentially with $$n$$, but am seeking an analytical solution. Extensive googling hasn't got me there, and neither has my sub-par knowledge of linear algebra.

Apologies if this is really basic, and thanks for the help!

• What happens asymptotically depends fundamentally on how you modify $\mathbf X$ as $n$ increases. What do you propose?
– whuber
Commented Dec 2, 2019 at 20:29

If you have a linear model such as: $$\mathbf{y} = \mathbf{X} \mathbf{p} + \mathbf{e},$$ where $$\mathbf{X}$$ is known and $$\mathbf{e}$$ is zero mean with covariance $$\mathbf{C_e}$$ (other than that, the pdf of $$\mathbf{e}$$ is arbitrary), then according to the Gauss-Markov theorem the Best Unbiased Linear Estimator (BLUE) of $$\mathbf{p}$$ is $$\hat{\mathbf{p}} = (\mathbf{X}^\top\mathbf{C_e}^{-1}\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{C_e}^{-1}\mathbf{y},$$ with covariance $$\mathbf{C}_{\hat{\mathbf{p}}} = (\mathbf{X}^\top\mathbf{C_e}^{-1}\mathbf{X})^{-1}.$$ So if we want to know how the variance of $$\hat{\mathbf{p}}$$ depends on the sample number, we can make life simple and assume $$p$$ is a simple scalar, $$\mathbf{X} = \mathbf{1}_N$$ ($$\mathbf{1}_N$$ is a column vector of size N) and $$\mathbf{C_e} = \sigma^2\mathbf{I}$$. This corresponds to the model $$y_n = p + e_n$$ with $$n = 1\dots N$$ samples and $$e_n$$ is zero mean with variance $$\sigma^2$$. Then the covariance of $$\hat{p}$$ evaluates to $$\mathbf{C}_{\hat{\mathbf{p}}} = (\frac{1}{\sigma^2}\mathbf{1}_N^\top\mathbf{I}\mathbf{1}_N)^{-1} = \frac{\sigma^2}{N} = \text{var}(\hat{p}).$$ This can be extended to other models (other $$\mathbf{X}$$, $$\mathbf{p}$$ and $$\mathbf{C}_{\mathbf{e}}$$), but the result holds that the variance decreases as $$1/N$$. So if your variance decreased exponentially numerically, then something is wrong (assuming that you used the BLUE estimator).

More can be found for example in: Kay, S. M. (1993). Fundamentals of Statistical Signal Processing: Estimation Theory. In Englewood Cliffs NJ Prentice Hall (1st ed.).

• Thanks so much hakanc ! Commented Dec 4, 2019 at 4:46

Start off simple. Start with $$y_i = \beta_0 + \beta_1x_i + \epsilon_i$$. The regression line is given by $$E(y_i) = \beta_0 + \beta_1x_{i}$$. Suppose you have $$n$$ observations (i.e. a sample size of $$n$$). Then for each of the $$n$$ observations you would have (omitting the expectation operator):

\begin{align} y_1 &= \beta_0 + \beta_1x_{1}\\ y_2 &= \beta_0 + \beta_1x_{2}\\ \vdots \\ y_n &= \beta_0 + \beta_1x_{n}\\ \end{align}

and in matrix form you have

$$\underbrace{\begin{bmatrix} y_1 \\ y_2 \\ \vdots\\ y_n \end{bmatrix}}_{Y} = \underbrace{\begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots\\ 1 & x_n \end{bmatrix} \begin{bmatrix} \beta_1 \\ \beta_2 \end{bmatrix}}_{X\beta}.$$

The covariance is given by $$\sigma^2(X^TX)^{-1}$$. Can you take it from here?

• Thank you! This is so obvious and everything makes sense now. Giving the answer to the other person though, as it's more complete... sorry! Commented Dec 4, 2019 at 4:45