Let's say we want to train a convolutional neural network, what gradient descent method works better?
- Batch gradient descent
- Stochastic gradient descent
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
Stochastic gradient descent (SGD) is a simple optimization method for Batch gradient descent (BGD). The idea in SGD that you divide your all training data set in mini-batches and train them. SGD works worse than BGD but it's much faster because your weight update computation is easier.
More information about SGD you can read in online book: http://neuralnetworksanddeeplearning.com/chap2.html
For example if you have 1000 units data set in BGD you will compute gradient for all input unit and sum them all, but for SGD you separate input units to mini-batches. For example youк mini-batch will be 10 units. so you get from 1 to 10 units and learn your network on this samples. For the next iteration you get 11-20 input units and also train your network (but on this step you will have updated weights after previous mini-batch learning). If min-batch size equal to count of input units, so SGD is the same as BGD, for another cases they are different.
Hope this is more clear for you.
BGD is, theoretically, the best method. But in practice, as the computational power is limited, people tend to use SGD or mini-batch as compromise. If your computer/remote server is really powerful, BGD or limited-BFGS are the optimization methods you should choose.
According to http://cs231n.github.io/optimization-1/, usually, people use mini-batch gradient descent, with the batch size being 32, 64, 128, 256, 512, etc, depending on your computational capability and the problem. the mini-batch gradient usually outperforms BGD or SGD.