Let's say we want to train a convolutional neural network, what gradient descent method works better?
- Batch gradient descent
- Stochastic gradient descent
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
Stochastic gradient descent (SGD) is a simple optimization method for Batch gradient descent (BGD). The idea in SGD that you divide your all training data set in mini-batches and train them. SGD works worse than BGD but it's much faster because your weight update computation is easier.
More information about SGD you can read in online book: http://neuralnetworksanddeeplearning.com/chap2.html
UPDATE:
For example if you have 1000 units data set in BGD you will compute gradient for all input unit and sum them all, but for SGD you separate input units to mini-batches. For example youк mini-batch will be 10 units. so you get from 1 to 10 units and learn your network on this samples. For the next iteration you get 11-20 input units and also train your network (but on this step you will have updated weights after previous mini-batch learning). If min-batch size equal to count of input units, so SGD is the same as BGD, for another cases they are different.
Hope this is more clear for you.
-
$\begingroup$ thanks for the information. I still don't quite understand. I thought SGD and BGD are different things, isn't it? $\endgroup$ Jan 27, 2015 at 18:25
-
$\begingroup$ yes. For example if you have 1000 units data set in BGD you will compute gradient for all input unit and sum them all, but for SGD you separate input units to mini-batches. For example you mini-batch will be 10 units. so you get from 1 - to 10 units and learn your network on this samples. for the next iteration you get 11-20 input units and also train your network (but on this step you will have updated weights after previous mini-batch learning). If min-batch size equal to count of input units, so SGD is the sa,e as BGD, for another cases they are different. Hope this is more clear for you. $\endgroup$– itdxerJan 27, 2015 at 18:32
-
$\begingroup$ Interesting. if the min-batch size is equal to the count of input units, then SGD is the same as BGD. Then why would we want BGD? $\endgroup$ Jan 27, 2015 at 18:41
-
$\begingroup$ BGD is better than SGD, but if you have millions input units - learning will be executed much more time for BGD than for SGD. SGD - is just optimization trick for BGD $\endgroup$– itdxerJan 27, 2015 at 18:44
-
$\begingroup$ I think in SGD, you only use 1 sample at a time. See this: cs231n.github.io/optimization-1 $\endgroup$ Mar 4, 2015 at 22:17
$\begingroup$
$\endgroup$
3
BGD is, theoretically, the best method. But in practice, as the computational power is limited, people tend to use SGD or mini-batch as compromise. If your computer/remote server is really powerful, BGD or limited-BFGS are the optimization methods you should choose.
According to http://cs231n.github.io/optimization-1/, usually, people use mini-batch gradient descent, with the batch size being 32, 64, 128, 256, 512, etc, depending on your computational capability and the problem. the mini-batch gradient usually outperforms BGD or SGD.
-
2$\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. $\endgroup$ Nov 2, 2016 at 21:23
-
$\begingroup$ Another useful way you could expand on this would be to cite a supporting source, if you can find one. $\endgroup$ Nov 2, 2016 at 22:03
-