computation of generalization error

Question

I was following the course of Machine Learning of the University of Washington in Coursera, and there was a point that captured my attention. The material was about the Generalization error and the example was a classic one of estimating the price of a house given the square feet as a feature. In one part the lecturer says

in contrast to training error we can't actually compute generalization error. Because everything was relative to this true distribution, the true way in which the world works

I mean, is it really possible not to compute the generalization error? I wonder if this has to be with the bias and variance trade off, but is it really non-computable this error? why we cannot threat it like an optimization problem? Thanks for your help in advance.

Answer 1

You can estimate it by setting aside a completely independent test data set (which is difficult to achieve in practice) and use it just once, namely to estimate the final model's generalization error.

As soon as you use insights from the test data set to further optimize your model, then information has leaked and the test data set is not anymore independent. In that case, your estimate of generalization error would get biased.