I'm trying to train a binary classifier with Keras. This is my model:

model = Sequential()
model.add(Dense(DENSE_DIM, activation='relu', input_shape=(data_shape[1:])))
model.add(Dense(target_len, activation='softmax'))

adam = Adam(lr=learning_rate)
model.compile(loss='binary_crossentropy', optimizer=adam, metrics=['binary_accuracy'])

I train the model with the following line:

 model.fit(train, train_target, epochs=num_epochs, batch_size=batch_size, class_weight=weights, validation_data=(dev,dev_target))

I've removed the dropout from the hidden layer to the output layer and I have already checked the input and it is the same every time I run the model. Still I am getting slightly different accuracy and f1 measures:

TrainsetLen:(1231, 865) #EPOCH: (50, 'bert', 0.001, 64) AVG: 0.7047619047619048 F1: 0.693599566738888
TrainsetLen:(1231, 865) #EPOCH: (50, 'bert', 0.001, 64) AVG: 0.7180952380952381 F1: 0.7101035060334213

This is how I compute them:

fold_acc = accuracy_score(test_target, test_preds)
fold_f1 = f1_score(test_target, test_preds, average='macro')
print("Test Accuracy on fold "+str(fold)+": ",fold_acc)
print("Test F1 on fold "+str(fold)+": ",fold_f1)
avg_acc += fold_acc/num_folds
avg_f1 += fold_f1/num_folds

the order of the folds is also the same. Shouldn't I be getting consistent results? What can be causing inconsistency on my model?

Thanks for your time and attention, Lucas.


1 Answer 1

  1. The model initialization is random. Each update depends on the current position, which depends on the previous position and so to the initialization. Using well-tuned gradient descent on a strongly convex problem should remove this, but neural networks are not convex and gradient descent is hard to tune.
  2. Minibatch-based training is stochastic, because different minibatches are passed to the model. This is because minibatches are constructed by sampling without replacement within each epoch. Different minibatches imply different model predictions, which implies different losses and different gradients. So the updates for 1 epoch applied to the same data set can still be different, because at each step, the direction of the update will probably be different.

If you need repeatable output for each call to fit, you'll need to take some additional steps. The documentation explains how to do this.

  • $\begingroup$ Thanks for your answer. Is there a way that I can pass a seed for the random initialization of the model? I need to benchmark and test some different setups on the same model and the comparison becomes irrelevant when the model is not consistent. $\endgroup$ Jun 8, 2020 at 14:43
  • $\begingroup$ Whenever I have questions about how to use software, I find it helpful to read the documentation. keras.io/getting_started/faq/… $\endgroup$
    – Sycorax
    Jun 8, 2020 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.