Return to Answer

Stochastic Gradient Descent(SGD):

• can do gradient descent with every single sample of a whole dataset one sample by one sample, taking the same number of steps as the samples of a whole dataset in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens 100 times in one epoch which means model's parameters are updated 100 times in one epoch.

• 's pros:

  • It's good at a large dataset because it takes small memory not slowing down the computation.
  • It's good at online learning.
  • It doesn't need the repreparation of a whole dataset if you want to update a model.
  • It less gets stuck in local minima or saddle points than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD). .

• 's cons:

  • The computation is less stable than BGD and MBGD.
  • It's less strong in noise(noisy data) than BGD and MBGD.
  • It gets a less accurate value than BGD and MBGD.

Stochastic Gradient Descent(SGD):

• can do gradient descent with every single sample of a whole dataset one sample by one sample, taking the same number of steps as the samples of a whole dataset in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens 100 times in one epoch which means model's parameters are updated 100 times in one epoch.

• 's pros:

  • It's good at a large dataset because it takes small memory not slowing down the computation.
  • It's good at online learning.
  • It doesn't need the repreparation of a whole dataset if you want to update a model.
  • It can more easily escape local minima or saddle points than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD). .

• 's cons:

  • The computation is less stable than BGD and MBGD.
  • It's less strong in noise(noisy data) than BGD and MBGD.
  • It gets a less accurate value than BGD and MBGD.

Stochastic Gradient Descent(SGD):

• can do gradient descent with every single sample of a whole dataset one sample by one sample, taking the same number of steps as the samples of a whole dataset in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens 100 times in one epoch which means model's parameters are updated 100 times in one epoch.
• 's pros:
  • It's good at a large dataset because it takes small memory not slowing down the computation.
  • It's good at online learning.
  • The repreparation of a whole dataset is not needed if you want to update a model.
  • It less gets stuck in local minima or saddle points than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD).
• 's cons:
  • The computation is less stable being less strong in noise(noisy data) than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD). *Noise(noisy data) means outliers and anomalies.

Stochastic Gradient Descent(SGD):

• can do gradient descent with every single sample of a whole dataset one sample by one sample, taking the same number of steps as the samples of a whole dataset in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens 100 times in one epoch which means model's parameters are updated 100 times in one epoch.

• 's pros:

  • It's good at a large dataset because it takes small memory not slowing down the computation.
  • It's good at online learning.
  • It doesn't need the repreparation of a whole dataset if you want to update a model.
  • It less gets stuck in local minima or saddle points than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD). .

• 's cons:

  • The computation is less stable than BGD and MBGD.
  • It's less strong in noise(noisy data) than BGD and MBGD.
  • It gets a less accurate value than BGD and MBGD.

Stochastic Gradient Descent(SGD):

• can do gradient descent with every single sample of a whole dataset one sample by one sample, taking the same number of steps as the samples of a whole dataset in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens 100 times in one epoch which means model's parameters are updated 100 times in one epoch.
• 's pros:
  • It's good at a large dataset because it takes small memory not slowing down the computation.
  • It's good at online learning.
  • The repreparation of a whole dataset is not needed if you want to update a model.
  • It less gets stuck in local minima or saddle points than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD).
• 's cons:
  • The computation is less stable being less strong in noise(noisy data) than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD). *Noise(noisy data) means outliers and anomalies.

Stochastic Gradient Descent(SGD):

• can do gradient descent with every single sample of a whole dataset one sample by one sample, taking the same number of steps as the samples of a whole dataset in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens 100 times in one epoch which means model's parameters are updated 100 times in one epoch.
• 's pros:
  • It's good at a large dataset because it takes small memory not slowing down the computation.
  • It's good at online learning.
  • The repreparation of a whole dataset is not needed if you want to update a model.
  • It less gets stuck in local minima or saddle points than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD).
• 's cons:
  • The computation is less stable being less strong in noise(noisy data) than Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD). *Noise(noisy data) means outliers and anomalies.