Keras single hidden layer model outputting different results given same input?

Question

I'm trying to train a binary classifier with Keras. This is my model:

model = Sequential()
model.add(Dense(DENSE_DIM, activation='relu', input_shape=(data_shape[1:])))
#model.add(Dropout(0.3))
model.add(Dense(target_len, activation='softmax'))
model.summary()

adam = Adam(lr=learning_rate)
model.compile(loss='binary_crossentropy', optimizer=adam, metrics=['binary_accuracy'])

I train the model with the following line:

 model.fit(train, train_target, epochs=num_epochs, batch_size=batch_size, class_weight=weights, validation_data=(dev,dev_target))

I've removed the dropout from the hidden layer to the output layer and I have already checked the input and it is the same every time I run the model. Still I am getting slightly different accuracy and f1 measures:

TrainsetLen:(1231, 865) #EPOCH: (50, 'bert', 0.001, 64) AVG: 0.7047619047619048 F1: 0.693599566738888
TrainsetLen:(1231, 865) #EPOCH: (50, 'bert', 0.001, 64) AVG: 0.7180952380952381 F1: 0.7101035060334213

This is how I compute them:

fold_acc = accuracy_score(test_target, test_preds)
fold_f1 = f1_score(test_target, test_preds, average='macro')
print("Test Accuracy on fold "+str(fold)+": ",fold_acc)
print("Test F1 on fold "+str(fold)+": ",fold_f1)
avg_acc += fold_acc/num_folds
avg_f1 += fold_f1/num_folds

the order of the folds is also the same. Shouldn't I be getting consistent results? What can be causing inconsistency on my model?

Thanks for your time and attention, Lucas.

Answer 1

1. The model initialization is random. Each update depends on the current position, which depends on the previous position and so to the initialization. Using well-tuned gradient descent on a strongly convex problem should remove this, but neural networks are not convex and gradient descent is hard to tune.
2. Minibatch-based training is stochastic, because different minibatches are passed to the model. This is because minibatches are constructed by sampling without replacement within each epoch. Different minibatches imply different model predictions, which implies different losses and different gradients. So the updates for 1 epoch applied to the same data set can still be different, because at each step, the direction of the update will probably be different.

If you need repeatable output for each call to fit, you'll need to take some additional steps. The documentation explains how to do this.