It is mentioned in the ESL book Page 19, that Knn and Least squares end up approximating conditional expectation by averages.
To explain the statements in detail, the book mentions 3 equations.
- The one below is the matrix notation of the Least squares equation, after derivating w.r.t Beta. (eq: 2.6)
- The second equation is obtained after assuming . This is then subsituted in the equation and then differentiating, we get the below equation (eq: 2.16)
For the above equation the author says, "Note we have not conditioned on X; rather we have used our knowledge of the functional relationship to pool over values of X. The least squares solution (2.6) amounts to replacing the expectation in (2.16) by averages over the training data. So both k-nearest neighbors and Least squares end up approximating conditional expectations by averages."
After going through this, I am unable to intuitively understand how the similarities hold between them. Can somebody please explain.