Gradient Descent (GD) vs Stochastic Gradient Descent (SGD)

Question

I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct.

My understanding of the difference between gradient descent (GD) and stochastic gradient descent (SGD) is:

1. In Gradient Descent (GD), we perform the forward pass using ALL the train data before starting the backpropagation pass to adjust the weights. This is called (one epoch).
2. In Stochastic Gradient Descent (SGD), we perform the forward pass using a SUBSET of the train set followed by backpropagation to adjust the weights. Hence, this is called (one iteration).

Is that correct?

Answer 1

Gradient Descent is an iterative method to solve the optimization problem. There is no concept of "epoch" or "batch" in classical gradient decent. The key of gradient decent are

• Update the weights by the gradient direction.
• The gradient is calculated precisely from all the data points.

Stochastic Gradient Descent can be explained as: quick and dirty way to "approximate gradient" from one single data point. If we relax on this "one single data point" to "a subset of data", then the concepts of batch and epoch come.

I have a related answer here (with code and plot for the demo)

How could stochastic gradient descent save time comparing to standard gradient descent?