0
$\begingroup$

I know this question is asked billion times, but I could not really find an answer to my situation. So, I want to show all the logs of Keras model learning. The problem is I don't know if my model is overfitting or not. Because both training and validation loss gets better and validation accuracy floats around 76%. So please tell me if there's a problem based on what you can see from logs. Thank you beforehand :)

Train on 12591 samples, validate on 1400 samples
Epoch 1/100
12591/12591 [==============================] - 224s 18ms/step - loss: 16.4598 - acc: 0.2811 - val_loss: 14.5194 - val_acc: 0.6200
Epoch 2/100
12591/12591 [==============================] - 208s 17ms/step - loss: 14.8376 - acc: 0.5306 - val_loss: 14.1055 - val_acc: 0.6636
Epoch 3/100
12591/12591 [==============================] - 208s 16ms/step - loss: 14.2127 - acc: 0.6126 - val_loss: 13.7273 - val_acc: 0.6750
Epoch 4/100
12591/12591 [==============================] - 206s 16ms/step - loss: 13.6790 - acc: 0.6515 - val_loss: 13.2612 - val_acc: 0.6800
Epoch 5/100
12591/12591 [==============================] - 206s 16ms/step - loss: 13.1478 - acc: 0.6739 - val_loss: 12.7899 - val_acc: 0.6979
Epoch 6/100
12591/12591 [==============================] - 204s 16ms/step - loss: 12.6308 - acc: 0.6889 - val_loss: 12.3134 - val_acc: 0.7021
Epoch 7/100
12591/12591 [==============================] - 204s 16ms/step - loss: 12.1130 - acc: 0.7067 - val_loss: 11.8419 - val_acc: 0.7107
Epoch 8/100
12591/12591 [==============================] - 204s 16ms/step - loss: 11.6277 - acc: 0.7235 - val_loss: 11.3987 - val_acc: 0.7129
Epoch 9/100
12591/12591 [==============================] - 203s 16ms/step - loss: 11.1752 - acc: 0.7381 - val_loss: 11.0041 - val_acc: 0.7236
Epoch 10/100
12591/12591 [==============================] - 203s 16ms/step - loss: 10.7591 - acc: 0.7520 - val_loss: 10.6476 - val_acc: 0.7236
Epoch 11/100
12591/12591 [==============================] - 203s 16ms/step - loss: 10.3873 - acc: 0.7679 - val_loss: 10.3320 - val_acc: 0.7329
Epoch 12/100
12591/12591 [==============================] - 203s 16ms/step - loss: 10.0425 - acc: 0.7861 - val_loss: 10.0426 - val_acc: 0.7436
Epoch 13/100
12591/12591 [==============================] - 204s 16ms/step - loss: 9.7027 - acc: 0.8053 - val_loss: 9.7588 - val_acc: 0.7500
Epoch 14/100
12591/12591 [==============================] - 203s 16ms/step - loss: 9.4138 - acc: 0.8196 - val_loss: 9.5576 - val_acc: 0.7543
Epoch 15/100
12591/12591 [==============================] - 204s 16ms/step - loss: 9.1276 - acc: 0.8440 - val_loss: 9.3711 - val_acc: 0.7479
Epoch 16/100
12591/12591 [==============================] - 204s 16ms/step - loss: 8.8549 - acc: 0.8695 - val_loss: 9.3385 - val_acc: 0.7357
Epoch 17/100
12591/12591 [==============================] - 204s 16ms/step - loss: 8.6307 - acc: 0.8829 - val_loss: 9.1169 - val_acc: 0.7550
Epoch 18/100
12591/12591 [==============================] - 204s 16ms/step - loss: 8.4178 - acc: 0.9029 - val_loss: 8.9277 - val_acc: 0.7736
Epoch 19/100
12591/12591 [==============================] - 204s 16ms/step - loss: 8.2176 - acc: 0.9168 - val_loss: 8.7775 - val_acc: 0.7607
Epoch 20/100
12591/12591 [==============================] - 203s 16ms/step - loss: 8.0370 - acc: 0.9276 - val_loss: 8.7633 - val_acc: 0.7521
Epoch 21/100
12591/12591 [==============================] - 204s 16ms/step - loss: 7.8535 - acc: 0.9415 - val_loss: 8.5871 - val_acc: 0.7600
Epoch 22/100
12591/12591 [==============================] - 204s 16ms/step - loss: 7.6730 - acc: 0.9528 - val_loss: 8.5887 - val_acc: 0.7621
Epoch 23/100
12591/12591 [==============================] - 203s 16ms/step - loss: 7.5194 - acc: 0.9548 - val_loss: 8.4273 - val_acc: 0.7650
Epoch 24/100
12591/12591 [==============================] - 203s 16ms/step - loss: 7.3572 - acc: 0.9606 - val_loss: 8.4304 - val_acc: 0.7579
Epoch 25/100
12591/12591 [==============================] - 203s 16ms/step - loss: 7.2006 - acc: 0.9666 - val_loss: 8.4116 - val_acc: 0.7607
Epoch 26/100
12591/12591 [==============================] - 203s 16ms/step - loss: 7.0491 - acc: 0.9693 - val_loss: 8.0627 - val_acc: 0.7700
Epoch 27/100
12591/12591 [==============================] - 203s 16ms/step - loss: 6.9009 - acc: 0.9720 - val_loss: 8.4484 - val_acc: 0.7414
Epoch 28/100
12591/12591 [==============================] - 203s 16ms/step - loss: 6.7558 - acc: 0.9759 - val_loss: 8.1282 - val_acc: 0.7493
Epoch 29/100
12591/12591 [==============================] - 203s 16ms/step - loss: 6.6149 - acc: 0.9767 - val_loss: 7.7018 - val_acc: 0.7700
Epoch 30/100
12591/12591 [==============================] - 203s 16ms/step - loss: 6.4743 - acc: 0.9796 - val_loss: 7.6717 - val_acc: 0.7743
Epoch 31/100
12591/12591 [==============================] - 204s 16ms/step - loss: 6.3378 - acc: 0.9805 - val_loss: 7.8323 - val_acc: 0.7571
Epoch 32/100
12591/12591 [==============================] - 203s 16ms/step - loss: 6.2046 - acc: 0.9826 - val_loss: 7.5521 - val_acc: 0.7700
Epoch 33/100
12591/12591 [==============================] - 204s 16ms/step - loss: 6.0787 - acc: 0.9840 - val_loss: 7.5302 - val_acc: 0.7607
Epoch 34/100
12591/12591 [==============================] - 203s 16ms/step - loss: 5.9482 - acc: 0.9844 - val_loss: 7.2903 - val_acc: 0.7643
Epoch 35/100
12591/12591 [==============================] - 203s 16ms/step - loss: 5.8277 - acc: 0.9860 - val_loss: 7.2471 - val_acc: 0.7600
Epoch 36/100
12591/12591 [==============================] - 203s 16ms/step - loss: 5.7067 - acc: 0.9866 - val_loss: 6.9579 - val_acc: 0.7650
Epoch 37/100
12591/12591 [==============================] - 203s 16ms/step - loss: 5.5880 - acc: 0.9881 - val_loss: 6.9856 - val_acc: 0.7700
Epoch 38/100
12591/12591 [==============================] - 203s 16ms/step - loss: 5.4676 - acc: 0.9896 - val_loss: 6.6924 - val_acc: 0.7857
Epoch 39/100
12591/12591 [==============================] - 204s 16ms/step - loss: 5.3697 - acc: 0.9866 - val_loss: 6.6368 - val_acc: 0.7700
Epoch 40/100
12591/12591 [==============================] - 203s 16ms/step - loss: 5.2597 - acc: 0.9891 - val_loss: 6.6382 - val_acc: 0.7629
Epoch 41/100
12591/12591 [==============================] - 203s 16ms/step - loss: 5.1530 - acc: 0.9911 - val_loss: 6.5930 - val_acc: 0.7614
Epoch 42/100
12591/12591 [==============================] - 203s 16ms/step - loss: 5.0562 - acc: 0.9897 - val_loss: 6.2574 - val_acc: 0.7736
Epoch 43/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.9585 - acc: 0.9913 - val_loss: 6.1675 - val_acc: 0.7721
Epoch 44/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.8703 - acc: 0.9915 - val_loss: 6.0677 - val_acc: 0.7721
Epoch 45/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.7788 - acc: 0.9923 - val_loss: 6.0391 - val_acc: 0.7650
Epoch 46/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.6938 - acc: 0.9921 - val_loss: 5.8575 - val_acc: 0.7743
Epoch 47/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.6054 - acc: 0.9927 - val_loss: 6.0495 - val_acc: 0.7443
Epoch 48/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.5148 - acc: 0.9932 - val_loss: 5.6903 - val_acc: 0.7714
Epoch 49/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.4329 - acc: 0.9940 - val_loss: 5.6729 - val_acc: 0.7743
Epoch 50/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.3470 - acc: 0.9954 - val_loss: 5.7399 - val_acc: 0.7614
Epoch 51/100
12591/12591 [==============================] - 204s 16ms/step - loss: 4.2694 - acc: 0.9940 - val_loss: 5.6004 - val_acc: 0.7543
Epoch 52/100
12591/12591 [==============================] - 204s 16ms/step - loss: 4.1866 - acc: 0.9960 - val_loss: 5.4398 - val_acc: 0.7593
Epoch 53/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.1108 - acc: 0.9949 - val_loss: 5.3087 - val_acc: 0.7807
Epoch 54/100
12591/12591 [==============================] - 203s 16ms/step - loss: 4.0413 - acc: 0.9948 - val_loss: 5.2749 - val_acc: 0.7721
Epoch 55/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.9727 - acc: 0.9958 - val_loss: 5.3999 - val_acc: 0.7664
Epoch 56/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.9057 - acc: 0.9959 - val_loss: 4.9687 - val_acc: 0.7786
Epoch 57/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.8346 - acc: 0.9961 - val_loss: 4.9840 - val_acc: 0.7771
Epoch 58/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.7554 - acc: 0.9972 - val_loss: 4.7741 - val_acc: 0.7800
Epoch 59/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.6814 - acc: 0.9974 - val_loss: 4.8041 - val_acc: 0.7729
Epoch 60/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.6093 - acc: 0.9970 - val_loss: 4.7305 - val_acc: 0.7657
Epoch 61/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.5415 - acc: 0.9978 - val_loss: 4.5445 - val_acc: 0.7736
Epoch 62/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.4665 - acc: 0.9975 - val_loss: 4.5500 - val_acc: 0.7771
Epoch 63/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.3990 - acc: 0.9975 - val_loss: 4.6668 - val_acc: 0.7629
Epoch 64/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.3276 - acc: 0.9984 - val_loss: 4.3481 - val_acc: 0.7650
Epoch 65/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.2518 - acc: 0.9985 - val_loss: 4.1594 - val_acc: 0.7829
Epoch 66/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.1838 - acc: 0.9980 - val_loss: 4.2286 - val_acc: 0.7836
Epoch 67/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.1161 - acc: 0.9990 - val_loss: 4.0789 - val_acc: 0.7693
Epoch 68/100
12591/12591 [==============================] - 203s 16ms/step - loss: 3.0450 - acc: 0.9984 - val_loss: 4.0762 - val_acc: 0.7650
Epoch 69/100
12591/12591 [==============================] - 203s 16ms/step - loss: 2.9828 - acc: 0.9983 - val_loss: 3.9904 - val_acc: 0.7657
Epoch 70/100
12591/12591 [==============================] - 203s 16ms/step - loss: 2.9209 - acc: 0.9986 - val_loss: 3.9936 - val_acc: 0.7607
Epoch 71/100
12591/12591 [==============================] - 203s 16ms/step - loss: 2.8661 - acc: 0.9982 - val_loss: 3.7985 - val_acc: 0.7886
Epoch 72/100
12591/12591 [==============================] - 204s 16ms/step - loss: 2.7895 - acc: 0.9988 - val_loss: 3.7508 - val_acc: 0.7743
Epoch 73/100
12591/12591 [==============================] - 203s 16ms/step - loss: 2.7246 - acc: 0.9987 - val_loss: 3.6916 - val_acc: 0.7786
Epoch 74/100
12591/12591 [==============================] - 203s 16ms/step - loss: 2.6647 - acc: 0.9991 - val_loss: 4.0022 - val_acc: 0.7571
Epoch 75/100
12591/12591 [==============================] - 204s 16ms/step - loss: 2.6087 - acc: 0.9988 - val_loss: 3.7985 - val_acc: 0.7686
Epoch 76/100
12591/12591 [==============================] - 203s 16ms/step - loss: 2.5521 - acc: 0.9991 - val_loss: 3.6268 - val_acc: 0.7786
Epoch 77/100
12591/12591 [==============================] - 204s 16ms/step - loss: 2.5099 - acc: 0.9986 - val_loss: 3.5174 - val_acc: 0.7800
Epoch 78/100
12591/12591 [==============================] - 203s 16ms/step - loss: 2.4618 - acc: 0.9986 - val_loss: 3.5259 - val_acc: 0.7729
Epoch 79/100
12591/12591 [==============================] - 203s 16ms/step - loss: 2.4179 - acc: 0.9991 - val_loss: 3.3663 - val_acc: 0.7857

Here's my model

from keras.models import Sequential
from keras.layers.core import Flatten, Dense, Dropout
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D, BatchNormalization, Dropout, Activation
from keras import regularizers

def get_model(data, classes):

    # L2 regularizer init
    # cnn one
    regularizer_cnn = regularizers.l2(0.001)
    # mlp one
    regularizer_mlp = regularizers.l2(0.001)

    model = Sequential()
    # layer 1
    model.add(ZeroPadding2D((1,1),input_shape=data.shape[1:]))
    model.add(Conv2D(64, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())

    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(64, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2,2), strides=(2,2)))

    # layer 2
    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(128, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())

    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(128, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2,2), strides=(2,2)))

    # layer 3
    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(256, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())

    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(256, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())

    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(256, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2,2), strides=(2,2)))

    # layer 4
    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(512, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())

    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(512, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())

    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(512, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2,2), strides=(2,2)))

    # layer 5
    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(512, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())

    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(512, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())

    model.add(ZeroPadding2D((1,1)))
    model.add(Conv2D(512, (3, 3), kernel_regularizer=regularizer_cnn))
    model.add(Activation('relu'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2,2), strides=(2,2)))

    # Fully connected layer
    model.add(Flatten())

    model.add(Dense(4096, kernel_regularizer=regularizer_mlp))
    model.add(Activation('relu'))
    model.add(Dropout(0.7))

    model.add(Dense(4096, kernel_regularizer=regularizer_mlp))
    model.add(Activation('relu'))
    model.add(Dropout(0.6))

    model.add(Dense(len(classes)))
    model.add(Activation('softmax'))

    return model

Additional stuff.

from keras.callbacks import ReduceLROnPlateau, ModelCheckpoint
from keras.optimizers import RMSprop, Adam
from sklearn.utils import class_weight

def get_optimizer():
  # optimizer
  return RMSprop( lr=0.00001, 
  #               rho=0.9, 
  #               epsilon=1e-08, 
                  decay=1e-6)

def get_callback_savepoint():
  return ModelCheckpoint('model-{epoch:03d}-{acc:03f}-{val_acc:03f}.h5',  
                         monitor='val_acc',
                         save_best_only=True, 
                         mode='max')

def get_callback_annealer():
  # Set a learning rate annealer
  return ReduceLROnPlateau(monitor='val_acc', 
                           patience=3, 
                           verbose=1, 
                           factor=0.5, 
                           min_lr=0.0001)

def get_weights(y_train):
  return class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)

And the model fitting part

model = get_model(X_train, classes)
# optimizer
opt = get_optimizer()

# Set a learning rate annealer
learning_rate_reduction = get_callback_annealer()
# checkpoint
checkpoint = get_callback_savepoint()

model.compile(loss='sparse_categorical_crossentropy',
              metrics=['accuracy'],
              optimizer=opt)

weights = get_weights(y_train)
print(weights)

model.fit(X_train, y_train, 
          batch_size=32, 
          epochs=100, 
          validation_data=(X_val, y_val),
          class_weight=weights,
          callbacks=[learning_rate_reduction, checkpoint])
$\endgroup$
0
$\begingroup$

One basic notion of identifying whether your model is overfitting is when there is a difference between the performance metric of the training set and validation set, in your case it's 99% ~ 76, which is quite massive.( considering that you have ruled out the possibility of high bias)

Moreover, you measure accuracy in your model hence if a data point has 0.51 probability & 0.99 probability of belonging to class 1 it won't make a difference in the accuracy metric. However, this makes the loss of your network decrease since your network is now leaning to get things which is already classified as "right" more "right". This is the reason for the decrease in your loss.

However, I feel accuracy is a decent enough metric if you are concerned with plain old classification and not probabilities. So you should always make a decision based on the metric that's more related to your problem.

It is expected that DNN's overfit quite easily hence you can try lowering learning rate, using dropout or batch norm.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the response @Axelius, I added my source code to the question, because I think I already did what you suggested. Do you spot anything suspicious? $\endgroup$ – Mensur Qulami Mar 11 '19 at 10:29
  • $\begingroup$ Is there class imbalance in your case or the distribution of labels different in train set & validation set? You could also try running for a few more epochs, sometimes it also helps $\endgroup$ – Axelius Mar 11 '19 at 10:33
  • $\begingroup$ To be honest, I didn't check that kind of imbalance. There was an imbalance in the number of classes but I managed to deal with it using data augmentation, but I actually splitted between validation and train data using train_test_split of sklearn which maybe I did wrong. After posting this, my validation accuracy now is 80% at best, and dropped a bit afterwards. Validation loss is currently 2.95. Currently I am running the CNN in Google's Colab and it is not that much faster (still million times faster than my laptop), so I cannot really run 100 epochs and run another 100 and check.Takes time $\endgroup$ – Mensur Qulami Mar 11 '19 at 10:39
  • $\begingroup$ Firstly always ensure that your metric takes class imbalance into account. Since you have ensured using data augmentation that class imbalance is not an issue then proceeding forward do a stratified split using sklearn since it ensures a similar distribution of classes in train & test. I would say increasing epochs is the last step to improve a model since sometimes the loss functions don't have a good landscape. $\endgroup$ – Axelius Mar 11 '19 at 14:18
  • $\begingroup$ I did that too, plus increased epochs to 500, it's on 23th but there's no change for now. Any other ideas? $\endgroup$ – Mensur Qulami Mar 11 '19 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.