What is Stochastic Gradient Descent (SGD) in simple terms?

Question

This year, I am doing a science fair project that involves deep learning and neural networks. When I write my report and present my project, I want to do so in very simple terms so that my teachers or classmates, who know absolutely nothing about neural networks, can understand. How can I explain Stochastic Gradient Descent(SGD) and parameter updates in a way anyone could understand?

Answer 1

I think you should explain (stochastic) gradient descent in terms of hill-climbing: We stand in a point on a hill and we want to select the path that guides us to the valley as fast as possible. Intuitively, we should select the direction that has the steepest slope. This is analogous to moving in the opposite direction of the gradient in the current point since gradient gives us the best direction upward. To find this direction, we can use all of our training data to check which direction has the most slope, but this is obviously time-consuming. Therefore, by stochastic gradient descent, we only consider one training sample and rely on that sample to tell us what direction has the most slope. In some points, we may select the wrong direction, since other training samples are neglected, but in the long run, we will eventually reach to the valley.

Answer 2

Stochastic Gradient Descent(SGD):

• can do gradient descent with every single sample of a whole dataset one sample by one sample, taking the same number of steps as the samples of a whole dataset in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens 100 times in one epoch which means model's parameters are updated 100 times in one epoch.

• 's pros:

  • It's good at a large dataset because it takes small memory not slowing down the computation.
  • It's good at online learning.
  • It doesn't need the repreparation of a whole dataset if you want to update a model.
  • It can more easily escape local minima or saddle points than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD). .

• 's cons:

  • The computation is less stable than BGD and MBGD.
  • It's less strong in noise(noisy data) than BGD and MBGD.
  • It gets a less accurate value than BGD and MBGD.