5
$\begingroup$

In Chapter 8 (Optimization for training deep models) page 278 of deep learning book (pdf: https://www.deeplearningbook.org/contents/optimization.html), it is stated that

An interesting motivation for minibatch stochastic gradient descent is that it follows the gradient of the true generalization error (equation 8.2) as long as no examples are repeated. Most implementations of minibatch stochastic gradient descent shuffle the dataset once and then pass through it multiple times. On the first pass, each minibatch is used to compute an unbiased estimate of the true generalization error. On the second pass, the estimate becomes biased because it is formed by resampling values that have already been used, rather than obtaining new fair samples from the data-generating distribution.

Why using the same data for more than one causes to have a biased estimate of the generalization error? What is the idea behind it? Do we assume that, in real life, it is not possible to have the same data point twice (or more), or there is something else?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ @MasA, read the long paragraph on page 277. I believe that the authors require that different samples be independent for the estimate to be unbiased. Using the same sample twice means that the two "samples", pass 1 and pass 2, are clearly not independant. $\endgroup$
    – Avraham
    Commented Dec 23, 2021 at 17:36
  • $\begingroup$ @Avraham thank you! I read this part. As far as I understood, if my data set is not very large (say 1000 examples) and I have a mini-batch size of 50, even if at each pass/iteration/epoch I shuffle the data before split it into mini-batches, since I will go through the same data points more than one time, generalization error estimate will be biased. $\endgroup$
    – Mas A
    Commented Dec 23, 2021 at 17:45
  • $\begingroup$ I think this answer is related: datascience.stackexchange.com/questions/82371/… , hope it helps $\endgroup$
    – Javier TG
    Commented Dec 24, 2021 at 1:07

1 Answer 1

Reset to default
1
$\begingroup$

Remember that generalization error is the expectation over data drawn from the true data distribution. If you're repeating examples from a finite sample, then you're replacing the true data distribution with the data in your sample.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thank you for pointing this out and adding to my question this part. Since in neural networks we use iterative methods for optimization, therefore, at the end of the day, in mini-batch GD (or full batch GD, SGD), gradients will never follow the true generalization error $\endgroup$
    – Mas A
    Commented Dec 24, 2021 at 11:24
  • $\begingroup$ So it is an unbiased estimator of the finite sample (training error) but a biased estimator of the true distribution (generalization error)? $\endgroup$
    – Antonios Sarikas
    Commented May 17 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.