5
$\begingroup$

In Chapter 8 (Optimization for training deep models) page 278 of deep learning book (pdf: https://www.deeplearningbook.org/contents/optimization.html), it is stated that

An interesting motivation for minibatch stochastic gradient descent is that it follows the gradient of the true generalization error (equation 8.2) as long as no examples are repeated. Most implementations of minibatch stochastic gradient descent shuffle the dataset once and then pass through it multiple times. On the first pass, each minibatch is used to compute an unbiased estimate of the true generalization error. On the second pass, the estimate becomes biased because it is formed by resampling values that have already been used, rather than obtaining new fair samples from the data-generating distribution.

Why using the same data for more than one causes to have a biased estimate of the generalization error? What is the idea behind it? Do we assume that, in real life, it is not possible to have the same data point twice (or more), or there is something else?

$\endgroup$
5
  • 1
    $\begingroup$ Please do not delete and re-post the same question. stats.stackexchange.com/questions/555288/… If you want to draw attention to a question, you can place a bounty on it. $\endgroup$
    – Sycorax
    Dec 23, 2021 at 17:08
  • $\begingroup$ I didn't know about bounty think but I will check it. What do you propose to me to get an answer to my question? Are there any problems with my question? I am open to your suggestions to improve my questions. $\endgroup$
    – Mas A
    Dec 23, 2021 at 17:26
  • 2
    $\begingroup$ @MasA, read the long paragraph on page 277. I believe that the authors require that different samples be independent for the estimate to be unbiased. Using the same sample twice means that the two "samples", pass 1 and pass 2, are clearly not independant. $\endgroup$
    – Avraham
    Dec 23, 2021 at 17:36
  • $\begingroup$ @Avraham thank you! I read this part. As far as I understood, if my data set is not very large (say 1000 examples) and I have a mini-batch size of 50, even if at each pass/iteration/epoch I shuffle the data before split it into mini-batches, since I will go through the same data points more than one time, generalization error estimate will be biased. $\endgroup$
    – Mas A
    Dec 23, 2021 at 17:45
  • $\begingroup$ I think this answer is related: datascience.stackexchange.com/questions/82371/… , hope it helps $\endgroup$
    – Javier TG
    Dec 24, 2021 at 1:07

1 Answer 1

1
$\begingroup$

Remember that generalization error is the expectation over data drawn from the true data distribution. If you're repeating examples from a finite sample, then you're replacing the true data distribution with the data in your sample.

$\endgroup$
1
  • $\begingroup$ thank you for pointing this out and adding to my question this part. Since in neural networks we use iterative methods for optimization, therefore, at the end of the day, in mini-batch GD (or full batch GD, SGD), gradients will never follow the true generalization error $\endgroup$
    – Mas A
    Dec 24, 2021 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.