# Tensorflow loss and accuracy during training weird values

I am doing some testing with tensorflow, and I bumbed into a very weird behaviour. Here is my code

fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images1, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images1[:32] / 255.0
train_labels = train_labels[:32]
test_images = test_images / 255.0

batch_size = 32
epochs     = 1

train_data = tf.data.Dataset.from_tensor_slices((train_images, train_labels)).batch(batch_size)

input_layer   = tf.keras.layers.Input(shape=(28,28,))
flatter       = tf.keras.layers.Flatten()(input_layer)
dense1        = tf.keras.layers.Dense(128,
kernel_regularizer=tf.keras.regularizers.l2(0.01),
activation='relu')(flatter)
dense2        = tf.keras.layers.Dense(64,
kernel_regularizer=tf.keras.regularizers.l2(0.01),
activation='relu')(dense1)
output_layer = tf.keras.layers.Dense(10,
activation='softmax',name='output')(dense2)
model_naive = tf.keras.models.Model(inputs=input_layer,outputs=output_layer)

loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
model_naive.summary()
history = model_naive.fit(x=train_data, validation_data=train_data, epochs=epochs)
model_naive.evaluate(train_data)


I simply import fashon_mnist (only one batch) and pass it as both train and validation data, to make comparison. No dropout in the network... so I would expect to find same loss and metric but this is the output

1/1 [==============================] - 1s 815ms/step - loss: 5.4532 - accuracy: 0.0312 - val_loss: 5.0106 - val_accuracy: 0.3125


To be sure I even did a model.evaluate() and this is what I find

1/1 [==============================] - 0s 12ms/step - loss: 5.0106 - accuracy: 0.3125


exactly the same found during training.

So, provided the evaluation is correct...what are these numbers "loss: 5.4532 - accuracy: 0.0312" ? I am using only one batch, so I would expect no averages over batches are involved.

EDIT: With only 1 batch keras weirdly seems to print loss and score before applying gradients. The same does not happen with more than one batch where probably some strange average is performed. Still not solved the issue, any comment is still very welcomed!

The first question I would like to ask, why you are using only 32 images from 60000 in the training set?

train_images = train_images1[:32] / 255.0
train_labels = train_labels[:32]


With such a small amount of data and only one training epoch, you hardly can expect anything meaningful. Moreover, your batch size the whole training data, I hardly see any way to split into train and validation in this case.

Loss is the computed value of SparseCategoricalCrossEntropy, which you pass as a loss function.

• You completely misunderstood my point! I am using 1 batch on purpose since I want to exclude any batch average during training. I am not using any splitting among train and validation (please read carefully model.fit). Don't look at the numbers... I am not training a model for usage, I am asking a question about the internal operations of tensorflow. What you should focus on is that those two lines are different loss: 5.4532 - accuracy: 0.0312 loss: 5.0106 - accuracy: 0.3125 and they should not. I repeat: no-droupout, only 1 batch, no validation set (validation=train).
– Dave
Feb 5, 2021 at 10:19
• @Dave my bad, I've looked throught the source code of Tensorflow and it seems to be ultimately no reason for the difference on training and validation Feb 5, 2021 at 13:08
• @Dave, this question is more appropriate in other sections of SE like Stack Overflow or Data Science, and even better for the Keras github. Feb 9, 2021 at 8:40