5
$\begingroup$

I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid function in last layer and two hidden layers, both with 400 units. NN is created using following Keras code:

model = Sequential()
model.add(Dense(units=139, input_dim=len(list(X))))

model.add(Dense(units=400))
model.add(Activation('relu'))
model.add(Dropout(0.2))

model.add(Dense(units=400))
model.add(Activation('relu'))
model.add(Dropout(0.2))

model.add(Dense(units=1))
model.add(Activation('sigmoid'))

model.compile(optimizer='sgd',
          loss='binary_crossentropy',
          metrics=['accuracy'])

checkpoint = ModelCheckpoint('tempmodelKeras.h5',period=1)
custom = LambdaCallback(on_epoch_end=lambda epoch,logs: test_callback_wrapper())
model.fit(X, y, epochs=500, batch_size=128, callbacks=[checkpoint,custom])

test_callback_wrapper() is just for testing model after each epoch on test dataset and then calculating average precision score for different thresholds.

Now the part I need to help with: Following images shows epoch number on X axis, and average precision on Y axis, for test set.

Average precision for different batch sizes during epochs

I tried to three different batch sizes (32, 128, 256), but if you look at the plot, it looks that smaller batch sizes are 'faster' in terms of number of epoch needed to reach maximum average precision, but are more prone to overfitting. However, I read several articles, where was written that larger batches usually leads to overfitting and smaller batches are better. How is it possible that in my image, it looks vice versa, that larger batches perform better?

$\endgroup$
  • $\begingroup$ Welcome to the site! I have to admit this is something new to me, intuitively I could not think of a reason why larger batches would result in overfitting. However there seems to be literature, as you point out, that reports this phenomenon. Have you seen this thread? Also you are on the verge of answering your own question, why not plot the in-sample results alongside out-of-sample and then you can update the question? $\endgroup$ – Zhubarb Apr 25 at 16:34
2
$\begingroup$

Using "epoch" on the x-axis to compare "speed" of convergence while comparing different batch sizes makes no sense since the number of weight updates per epoch depends on the batch size. When using smaller batches, the learning algorithm performs more weight updates per epoch and it naturally it seems to converge faster.

In your image, the gray curve (batch size = 32) starts decreasing (overfitting) around epoch 40. The orange curve (batch size = 128, four times more) seems to peak around epoch 160—four times later, which is exactly after the same number of weight updates. Possibly you might see similar peak on the blue curve around epoch 320.

Generally, smaller batches lead to noisier gradient estimates and are better capable to escape poor local minima and prevent overfitting. On the other hand, tiny batches may be too noisy for good learning. In the end, it is just another hyperparameter that one needs to tune on the specific dataset. In your case, it does not seem to play a big role, since all curves peak around the same accuracy, and early stopping should lead to the same performance regardless of the batch size.

$\endgroup$
  • $\begingroup$ Thanks. This helps me a lot. I understood that smaller batch sizes can be noisier, but this explains me why they reach peak faster and then starts overfitting. $\endgroup$ – Jan Musil Apr 26 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.