This is a follow-up question for this post

This question is related to what is written in 19 page of ESL.

We have two statements. The following is a quote from the post.

The one below is the matrix notation of the Least squares equation, after derivating w.r.t 𝛽. (eq: 2.6)

$\widehat{\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}\boldsymbol{y}$

The second equation is obtained after assuming $f(x)\approx x^{T}\beta$. This is then substituted in the equation EPE(𝑓)=E(𝑌−𝑓(𝑋))2 and then differentiating, we get the below equation (eq: 2.16)

$\beta =(E[X^{T}X])^{-1}E[XY]$

Regarding the two statements, the books says

The least squares solution (2.6) amounts to replacing the expectation in (2.16) by averages over the training data.

I am not clear abou what it means by saying replacing the expectation by averages over the training data. Where does the averages come from?


1 Answer 1


You can rewrite that formula for $\hat\beta$ as $$\hat\beta = \left(\frac{1}{n}\sum_{i=1}^n x_ix_i^T\right)^{-1}\left(\frac{1}{n}\sum_{i=1}^n x_i^Ty_i \right)$$ which is the formula for $\beta$ only with the expectations replaced by averages over the $n$ observations in the sample.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.