I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct.
My understanding of the difference between gradient descent (GD) and stochastic gradient descent (SGD) is:
- In Gradient Descent (GD), we perform the forward pass using ALL the train data before starting the backpropagation pass to adjust the weights. This is called (one epoch).
- In Stochastic Gradient Descent (SGD), we perform the forward pass using a SUBSET of the train set followed by backpropagation to adjust the weights. Hence, this is called (one iteration).
Is that correct?