# Why estimates becomes biased after second pass (from deep learning book of Ian GoodFellow)

In Chapter 8 (Optimization for training deep models) page 278 of deep learning book (pdf: https://www.deeplearningbook.org/contents/optimization.html), it is stated that

An interesting motivation for minibatch stochastic gradient descent is that it follows the gradient of the true generalization error (equation 8.2) as long as no examples are repeated. Most implementations of minibatch stochastic gradient descent shuﬄe the dataset once and then pass through it multiple times. On the ﬁrst pass, each minibatch is used to compute an unbiased estimate of the true generalization error. On the second pass, the estimate becomes biased because it is formed by resampling values that have already been used, rather than obtaining new fair samples from the data-generating distribution.

Why using the same data for more than one causes to have a biased estimate of the generalization error? What is the idea behind it? Do we assume that, in real life, it is not possible to have the same data point twice (or more), or there is something else?

• Please do not delete and re-post the same question. stats.stackexchange.com/questions/555288/… If you want to draw attention to a question, you can place a bounty on it.
– Sycorax
Dec 23, 2021 at 17:08
• I didn't know about bounty think but I will check it. What do you propose to me to get an answer to my question? Are there any problems with my question? I am open to your suggestions to improve my questions. Dec 23, 2021 at 17:26
• @MasA, read the long paragraph on page 277. I believe that the authors require that different samples be independent for the estimate to be unbiased. Using the same sample twice means that the two "samples", pass 1 and pass 2, are clearly not independant. Dec 23, 2021 at 17:36
• @Avraham thank you! I read this part. As far as I understood, if my data set is not very large (say 1000 examples) and I have a mini-batch size of 50, even if at each pass/iteration/epoch I shuffle the data before split it into mini-batches, since I will go through the same data points more than one time, generalization error estimate will be biased. Dec 23, 2021 at 17:45
• I think this answer is related: datascience.stackexchange.com/questions/82371/… , hope it helps Dec 24, 2021 at 1:07