2
$\begingroup$

I was following the course of Machine Learning of the University of Washington in Coursera, and there was a point that captured my attention. The material was about the Generalization error and the example was a classic one of estimating the price of a house given the square feet as a feature. In one part the lecturer says

in contrast to training error we can't actually compute generalization error. Because everything was relative to this true distribution, the true way in which the world works

I mean, is it really possible not to compute the generalization error? I wonder if this has to be with the bias and variance trade off, but is it really non-computable this error? why we cannot threat it like an optimization problem? Thanks for your help in advance.

$\endgroup$

1 Answer 1

4
$\begingroup$

You can estimate it by setting aside a completely independent test data set (which is difficult to achieve in practice) and use it just once, namely to estimate the final model's generalization error.

As soon as you use insights from the test data set to further optimize your model, then information has leaked and the test data set is not anymore independent. In that case, your estimate of generalization error would get biased.

$\endgroup$
1
  • $\begingroup$ thank you very much @Michael M for your answer, one question could you be so kind to explain about that leakage information? $\endgroup$
    – Layla
    Aug 16, 2020 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.