How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

Question

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. Here is what happens.

According to what I have understood from Chapters 3 and 7 of this book, Stochastic Gradient Descent works at the following way:

1. For instance, set epoch = 5
2. For each epoch, randomly select one data point from the available data set and do the forward propagation. Compute the estimated value of $y$ and the associated error $-$ which depends on the chosen loss function;
3. Do the back propagation and update the weights vector;
4. If you did not complete 5 iterations yet, go back to step 1..

In summary, if I set up epoch = 5, I will update the weights vector only 5 times $-$ considering a single data point at each one of them. Obviously, it only makes sense if I define the epoch arbitrarily large.

---

Now, according to what I have understood about the (Mini-) Batch Gradient Descent, we have the following situation.

1. For instance, imagine that we have a data set of size n = 950 and set epoch = 5 and batch_size = 100;
2. For each epoch, select the first 100 data points and perform the forward propagation. Computed the estimated value of $y$ and the associated error.
3. Do the back propagation and update the weights vector;
4. Select the next 100 data points (let's say, from 101 to 200) and do it again: perform the forward propagation, compute the error, perform the back propagation, updated the weights vector [$\cdots$]. Until you finally select the last 50 data points (let's say, from 901 to 950) and do all the required stuff;
5. If you did not complete 5 iterations yet, go back to step 1..

In summary, if I have n = 950 and set up epoch = 5 and batch_size = 100, I will update the weights vector $\lceil\frac{950}{100}\rceil \times 5 = $ 50 times.

Here, if I choose batch_size = n I will have the "Traditional" Gradient Descent.

---

Thus, the first question is: am I right? Considering what I have written so far.

If so, how to explain the following scenario?

By using Keras, I will try to construct an ANN (Artificial Neural Network) to classify the MNIST data set. Here is the code:

from tensorflow import keras

mnist = keras.datasets.mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train / 255
X_test  = X_test  / 255

model = keras.models.Sequential([
    keras.layers.Flatten(input_shape = (28, 28)),
    keras.layers.Dense(units = 128, activation = 'tanh'),
    keras.layers.Dense(units =  64, activation = 'tanh'),
    keras.layers.Dense(units =  32, activation = 'tanh'),
    keras.layers.Dense(units =  10, activation = 'sigmoid')
])

model.compile(optimizer = 'sgd',
              loss = 'sparse_categorical_crossentropy',
              metrics = ['accuracy'])

model.fit(X_train, y_train, epochs = 5)

I know that the chosen activation function are not the most common ones (instead, I could have used ReLU for the hidden layers and SOFTMAX for the output), but that is not the point of my question. So let's move on.

Here is the thing: Since I am using the Stochastic Gradient Descent optimizer (optimizer = 'sgd'), I should not be able to set a batch_size (actually, the only option would be batch_size = 1, if I am not wrong). However, it is perfectly fine if I try to set batch_size = 32 as a parameter for the fit() method:

model.fit(X_train, y_train, epochs = 5, batch_size = 32)

Things get worst when I realized that, if I manually set batch_size = 1 the fitting process takes much longer, which does not make any sense according to what I described as being the algorithm.

So, the second question is: what am I missing?

Thanks in advance.

Answer 1

As a term of art, the usage of "stochastic gradient descent" is inconsistent.

• In some sources, SGD is exclusively the case of using 1 observation randomly-chosen without replacement per epoch to update a model.
• In other sources, stochastic gradient descent refers to using a randomly-selected sample of observations for updating the model, of any size, including a mini-batch of size 1 as a special case.

I do not see any compelling reason to reserve the terminology SGD for the case of 1 observation, so the convention that I use here is that SGD can have mini-batches of any size $1 \le m < n$. The special case of $m = n$ is batch gradient descent, and is not stochastic because it does not have a randomized component depending on random selection of data used for an update.

More information: Stochastic gradient descent Vs Mini-batch size 1

---

Your understanding is not correct. An epoch is a pass through all $n$ data points. (See: Why do neural network researchers care about epochs?)

So if you have $n$ observations and $k$ epochs, then SGD with a mini-batch size of $m=1$ computes $n k$ updates, not $k$ updates. The data points are chosen randomly, without replacement at each of $n$ iterations during the $k$ epochs.

SGD mini-batch gradient descent with $1 < m < n$ data points chosen randomly without replacement to update the model. This means that $\lceil \frac{n}{m} \rceil k$ total updates are applied during $k$ epochs.*

Computation time is shorter for $m > 1$ because that involves $nk \ge \lceil \frac{n}{m} \rceil k$ updates, so fewer forward passes and backward passes in total. Vectorization and other computational tricks improve the scaling properties of the arithmetic, and therefore the time it takes to complete an epoch, even though all $n$ samples are moving through the model during an epoch in both cases.

*In some unusual cases, a researcher might want to leave off the last few observations if the final mini-batch would have size less than $m$. One example is training a model with online hard negative mining, because online hard negatives are order statistics of the mini-batch, and the distribution of order statistics changes with sample size.