1
$\begingroup$

According to Deep Learning (Ian Goodfellow and Yoshua Bengio and Aaron Courville,p.111, available online):

1.Assume that train and test set are indentically distributed(IID assumption).

2.Then:

''We sample the training set, then use it to choose the parameters to reduce training set error, then sample the test set. Under this process, the expected test error is greater than or equal to the expected value of training error"

Could you provide me with mathematical proof of this statement?

$\endgroup$
1
$\begingroup$

There is no such a proof. That's just an intuitive thing. Model typically predicts training samples better than test samples, because it learns from the training data and test data is just something that model hasn't seen before. It's possible that test error is lower than training error, especially in case if samples are small.

$\endgroup$
  • 1
    $\begingroup$ Are you certain? I mean, in this case we assume that training and test set are drawn from exactly the same distribution, so as a result it allows us to apply more theoretical than intuitive approach, like Chernoff Bound etc. cs229.stanford.edu/notes/cs229-notes4.pdf For instance assume the case that we draw 100 samples from some p(x,y) distribution, train model, and then we draw 100 samples from the same p(x,y) as a test set. $\endgroup$ – mokebe Nov 30 '16 at 14:37
  • $\begingroup$ Even though they are identically distributed, it's possible due to chance that it's easier for model to guess test samples than training samples. The less samples you have the more likely this event. $\endgroup$ – itdxer Dec 2 '16 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.