1
$\begingroup$

According to Deep Learning (Ian Goodfellow and Yoshua Bengio and Aaron Courville,p.111, available online):

1.Assume that train and test set are indentically distributed(IID assumption).

2.Then:

''We sample the training set, then use it to choose the parameters to reduce training set error, then sample the test set. Under this process, the expected test error is greater than or equal to the expected value of training error"

Could you provide me with mathematical proof of this statement?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

There is no such a proof. That's just an intuitive thing. Model typically predicts training samples better than test samples, because it learns from the training data and test data is just something that model hasn't seen before. It's possible that test error is lower than training error, especially in case if samples are small.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Are you certain? I mean, in this case we assume that training and test set are drawn from exactly the same distribution, so as a result it allows us to apply more theoretical than intuitive approach, like Chernoff Bound etc. cs229.stanford.edu/notes/cs229-notes4.pdf For instance assume the case that we draw 100 samples from some p(x,y) distribution, train model, and then we draw 100 samples from the same p(x,y) as a test set. $\endgroup$
    – mokebe
    Nov 30, 2016 at 14:37
  • $\begingroup$ Even though they are identically distributed, it's possible due to chance that it's easier for model to guess test samples than training samples. The less samples you have the more likely this event. $\endgroup$
    – itdxer
    Dec 2, 2016 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.