How does the least squares solution replace expectation by averages over the training data?

Question

This is a follow-up question for this post

This question is related to what is written in 19 page of ESL.

We have two statements. The following is a quote from the post.

The one below is the matrix notation of the Least squares equation, after derivating w.r.t 𝛽. (eq: 2.6)

$\widehat{\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}\boldsymbol{y}$

The second equation is obtained after assuming $f(x)\approx x^{T}\beta$. This is then substituted in the equation EPE(𝑓)=E(𝑌−𝑓(𝑋))2 and then differentiating, we get the below equation (eq: 2.16)

$\beta =(E[X^{T}X])^{-1}E[XY]$

Regarding the two statements, the books says

The least squares solution (2.6) amounts to replacing the expectation in (2.16) by averages over the training data.

I am not clear abou what it means by saying replacing the expectation by averages over the training data. Where does the averages come from?

Answer 1

You can rewrite that formula for $\hat\beta$ as $$\hat\beta = \left(\frac{1}{n}\sum_{i=1}^n x_ix_i^T\right)^{-1}\left(\frac{1}{n}\sum_{i=1}^n x_i^Ty_i \right)$$ which is the formula for $\beta$ only with the expectations replaced by averages over the $n$ observations in the sample.