Say we wanted to learn $$f_{\theta}(\pmb{x})=y$$, with a loss function $$L(f_{\theta}(\pmb{x}),y)$$. We often want to choose $$\theta$$ which minimises the empirical loss, as the exact loss isn't available to us. We calculate the empirical loss based on observed data $$(\pmb{x}_1,y_1),...,(\pmb{x}_m,y_m)$$.

We can minimise the loss/empirical loss by gradient descent, with a gradient descent update looking like $$\theta_n=\theta_{n-1}-\sum\limits_{i=1}^m\nabla_{\theta}L(f_{\theta}(\pmb{x}_i),y_i)|_{\theta=\theta_{n-1}}$$ at step $$n \geq 1$$.

My question is the following: at each step $$n$$ in gradient descent, do we need to sample new data to use in the gradient descent update, or can we just use the same data for each update?

Until I saw it done the other way, I thought it was best to just use the same dataset throughout. But maybe the other method is correct, as the overall goal should be to minimise the true loss and not just the estimate (just minimising the estimate might lead to overfitting). This could just be my ignorance of gradient descent methods applied to supervised learning, and I am talking about two different variants of gradient descent.

Thanks.

• For the second method, do you mean bootstrap sampling from training set with size m? Commented Mar 17, 2021 at 22:55
• @gunes It could be a bootstrapped sample. The paper doesn't specify if it's a completely new sample, or a bootstrapped sample. They just say "sample k data points and then perform a gradient descent update", quite frustrating! Commented Mar 17, 2021 at 22:59
• @gunes Doing further research has brought me to a method called "mini-batch gradient descent". But I also realize that the paper I am reading doesn't necessarily do this anyway (although it might, it is unclear, not that it matters). I will delete this post in a few hours. Commented Mar 17, 2021 at 23:23
• Was about to post an answer about mini-batch GD, but it seems you already found it. Commented Mar 17, 2021 at 23:26