I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large should the batch size be?


1 Answer 1


The "sample size" you're talking about is referred to as batch size, $B$. The batch size parameter is just one of the hyper-parameters you'll be tuning when you train a neural network with mini-batch Stochastic Gradient Descent (SGD) and is data dependent. The most basic method of hyper-parameter search is to do a grid search over the learning rate and batch size to find a pair which makes the network converge.

To understand what the batch size should be, it's important to see the relationship between batch gradient descent, online SGD, and mini-batch SGD. Here's the general formula for the weight update step in mini-batch SGD, which is a generalization of all three types. [2]

$$ \theta_{t+1} \leftarrow \theta_{t} - \epsilon(t) \frac{1}{B} \sum\limits_{b=0}^{B - 1} \dfrac{\partial \mathcal{L}(\theta, \textbf{m}_b)}{\partial \theta} $$

  1. Batch gradient descent, $B = |x|$
  2. Online stochastic gradient descent: $B = 1$
  3. Mini-batch stochastic gradient descent: $B > 1$ but $B < |x|$.

Note that with 1, the loss function is no longer a random variable and is not a stochastic approximation.

SGD converges faster than normal "batch" gradient descent because it updates the weights after looking at a randomly selected subset of the training set. Let $x$ be our training set and let $m \subset x$. The batch size $B$ is just the cardinality of $m$: $B = |m|$.

Batch gradient descent updates the weights $\theta$ using the gradients of the entire dataset $x$; whereas SGD updates the weights using an average of the gradients for a mini-batch $m$. (Using the average as opposed to a sum prevents the algorithm from taking steps that are too large if the dataset is very large. Otherwise, you would need to adjust your learning rate based on the size of the dataset.) The expected value of this stochastic approximation of the gradient used in SGD is equal to the deterministic gradient used in batch gradient descent. $\mathbb{E}[\nabla \mathcal{L}_{SGD}(\theta, \textbf{m})] = \nabla \mathcal{L}(\theta, \textbf{x})$.

Each time we take a sample and update our weights it is called a mini-batch. Each time we run through the entire dataset, it's called an epoch.

Let's say that we have some data vector $\textbf{x} : \mathbb{R}^D$, an initial weight vector that parameterizes our neural network, $\theta_0 : \mathbb{R}^{S}$, and a loss function $\mathcal{L}(\theta, \textbf{x}) : \mathbb{R}^{S} \rightarrow \mathbb{R}^{D} \rightarrow \mathbb{R}^S$ that we are trying to minimize. If we have $T$ training examples and a batch size of $B$, then we can split those training examples into C mini-batches:

$$ C = \lceil T / B \rceil $$

For simplicity we can assume that T is evenly divisible by B. Although, when this is not the case, as it often is not, proper weight should be assigned to each mini-batch as a function of its size.

An iterative algorithm for SGD with $M$ epochs is given below:

\begin{align*} t &\leftarrow 0 \\ \textrm{while } t &< M \\ \theta_{t+1} &\leftarrow \theta_{t} - \epsilon(t) \frac{1}{B} \sum\limits_{b=0}^{B - 1} \dfrac{\partial \mathcal{L}(\theta, \textbf{m}_b)}{\partial \theta} \\ t &\leftarrow t + 1 \end{align*}

Note: in real life we're reading these training example data from memory and, due to cache pre-fetching and other memory tricks done by your computer, your algorithm will run faster if the memory accesses are coalesced, i.e. when you read the memory in order and don't jump around randomly. So, most SGD implementations shuffle the dataset and then load the examples into memory in the order that they'll be read.

The major parameters for the vanilla (no momentum) SGD described above are:

  1. Learning Rate: $\epsilon$

I like to think of epsilon as a function from the epoch count to a learning rate. This function is called the learning rate schedule.

$$ \epsilon(t) : \mathbb{N} \rightarrow \mathbb{R} $$

If you want to have the learning rate fixed, just define epsilon as a constant function.

  1. Batch Size

Batch size determines how many examples you look at before making a weight update. The lower it is, the noisier the training signal is going to be, the higher it is, the longer it will take to compute the gradient for each step.

Citations & Further Reading:

  1. Introduction to Gradient Based Learning
  2. Practical recommendations for gradient-based training of deep architectures
  3. Efficient Mini-batch Training for Stochastic Optimization
  • 1
    $\begingroup$ For simplicity we can assume that D is evenly divisible by B. Don't you mean that T should be evenly divisible by B? $\endgroup$
    – Antoine
    Commented Mar 24, 2017 at 8:57
  • 6
    $\begingroup$ and to actually answer the OP's question, you can add B is typically chosen between 1 and a few hundreds, e.g. B = 32 is a good default value, with values above 10 taking advantage of the speed-up of matrix-matrix products over matrix-vector products. (from Bengio's 2012 paper) $\endgroup$
    – Antoine
    Commented Mar 24, 2017 at 9:44
  • $\begingroup$ @sabalaba Nice answer. But isn't it that in the equation "An iterative algorithm for SGD with M epochs is given below" we will update weight after running over each mini-batch. In other words, shouldn't there be another loop (over the C mini batches) inside the loop over the epoch i.e. while t < M $\endgroup$ Commented Aug 14, 2017 at 5:19
  • 5
    $\begingroup$ In statistics, a sample consists of multiple observations. So, sample size is correctly interpreted by statisticians. In computer science (particularly, in machine learning) a sample is a single observation and a batch is a collection of samples. It could be a bit confusing. A sample to statisticians is a batch to data scientists ;) Source: en.wikipedia.org/wiki/Sample_size_determination $\endgroup$ Commented Dec 28, 2017 at 18:08
  • $\begingroup$ "The batch size parameter is just one of the hyper-parameters you'll be tuning . . ." Wouldn't running multiple tuning models defeat the whole purpose of SGD in the first place? I think the OP is asking for a rule of thumb for sample size. If your data contains $n$ records and $m$ variables, what is a suitable sample size to adequately calculate the gradient? $\endgroup$
    – RobertF
    Commented Mar 23, 2018 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.