# How does the least squares solution replace expectation by averages over the training data?

This is a follow-up question for this post

This question is related to what is written in 19 page of ESL.

We have two statements. The following is a quote from the post.

The one below is the matrix notation of the Least squares equation, after derivating w.r.t 𝛽. (eq: 2.6)

$$\widehat{\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}\boldsymbol{y}$$

The second equation is obtained after assuming $$f(x)\approx x^{T}\beta$$. This is then substituted in the equation EPE(𝑓)=E(𝑌−𝑓(𝑋))2 and then differentiating, we get the below equation (eq: 2.16)

$$\beta =(E[X^{T}X])^{-1}E[XY]$$

Regarding the two statements, the books says

The least squares solution (2.6) amounts to replacing the expectation in (2.16) by averages over the training data.

I am not clear abou what it means by saying replacing the expectation by averages over the training data. Where does the averages come from?

You can rewrite that formula for $$\hat\beta$$ as $$\hat\beta = \left(\frac{1}{n}\sum_{i=1}^n x_ix_i^T\right)^{-1}\left(\frac{1}{n}\sum_{i=1}^n x_i^Ty_i \right)$$ which is the formula for $$\beta$$ only with the expectations replaced by averages over the $$n$$ observations in the sample.