# How do I interpret my validation and training loss curve if there is a large difference between the two which closes in sharply

my CNN is meant to classify an image as one out of around 30 categories. I am training on 6400 samples using a batch size of 128. I am using Keras/ Tensorflow Architecture is Conv + Batch Normalization + Convo + Batch Normalization + MaxPooling2D repeated 4 times. Using an ADAM optimizer gives me following loss curves, orange is validation blue is training

There is a huge gap between validation and training loss which closes in eventually. I then tried an SGD optimizer

Which looked better but I still didn’t understand why training set is able to learn so quickly and validation loss only decreases after awhile. I thought maybe its because my learning rate is too large, causing the training loss to plunge initially. I tried reducing learning rate.

But doesn't change much. I thought maybe its due to initial overfitting and that I am “lucky” to have the validation loss eventually find its way lower, so I tried adding dropouts to my model

I'm still getting the same issue. I also tried changing the momentum from 0.5 to 0.9

What is causing the shape of my loss curve to be like that and how do I tweak it so validation loss drops quicker in sync? I don’t understand why there is a large gap between the two which only closes in after a few epochs.

I'm reluctant to increase my data-set since I've seen other similar projects being done with similar size data. Input image pixels have already been standardized to between 0-1

Edit: I'm posting my architecture below just for more info if it helps, hope someone takes a look cause to me its a real head-scratcher. Using Python, keras

feeder_input = Input(shape=(50, 55, 3))

layer_1_CovNet = Convolution2D(64, (3, 3), activation="relu", padding="same")(feeder_input)
layer_1_CovNet = BatchNormalization()(layer_1_CovNet)
layer_2_CovNet = Convolution2D(64, (3, 3), activation="relu", padding="same")(layer_1_CovNet)
layer_2_CovNet = BatchNormalization()(layer_2_CovNet)
layer_3_CovNetPl = MaxPooling2D()(layer_2_CovNet)

layer_4_CovNet = Convolution2D(128, (3, 3), activation="relu", padding="same")(layer_3_CovNetPl)
layer_4_CovNet = BatchNormalization()(layer_4_CovNet)
layer_5_CovNet = Convolution2D(128, (3, 3), activation="relu", padding="same")(layer_4_CovNet)
layer_5_CovNet = BatchNormalization()(layer_5_CovNet)
layer_6_CovNetPl = MaxPooling2D()(layer_5_CovNet)

layer_7_CovNet = Convolution2D(256, (3, 3), activation="relu", padding="same")(layer_6_CovNetPl)
layer_7_CovNet = BatchNormalization()(layer_7_CovNet)
layer_8_CovNet = Convolution2D(256, (3, 3), activation="relu", padding="same")(layer_7_CovNet)
layer_8_CovNet = BatchNormalization()(layer_8_CovNet)
layer_9_CovNetPl = MaxPooling2D()(layer_8_CovNet)

layer_10_CovNet = Convolution2D(512, (3, 3), activation="relu", padding="same")(layer_9_CovNetPl)
layer_10_CovNet = BatchNormalization()(layer_10_CovNet)
layer_11_CovNet = Convolution2D(512, (3, 3), activation="relu", padding="same")(layer_10_CovNet)
layer_11_CovNet = BatchNormalization()(layer_11_CovNet)
layer_12_CovNetPl = MaxPooling2D()(layer_11_CovNet)

ConvNetFlatten = Flatten()(layer_12_CovNetPl)

encoder_dense1 = Dense(512, activation='relu', name='encoder_dense1')
encoder_dense1_output = encoder_dense1(ConvNetFlatten)

totext_dense1 = Dense(len(char_dict), activation='softmax', name='totext_dense1')
totext_output1 = totext_dense1(encoder_dense1_output)

model_train = Model(feeder_input, totext_output1)

– Jim
Mar 21 '18 at 16:47
• @Jim thanks for taking a look jim. X axis is number of epochs, Y axis is loss. Again the confusion is mainly why the training loss goes down so sharply and validation loss takes so long. The validation loss of the SGD optimization looks fine by itself, but again baffled why training loss is able to drop to <1 so quickly compared to validation. Is it cause of weight initialization? I feel like someone with more experience should be able to recognize and diagnose the problem quite easily Mar 22 '18 at 13:41
• When I look at the graphs, they raise two main questions: 1) How can the validation loss decrease and then increase again? 1.1) This behavior is present in some curves -- why? 2) What explains the sharp disparity between training loss and validation loss in all graphs?
– Jim
Apr 1 '18 at 18:28
• I'm curious to see this run with say, 50 - 100 epochs. Will it overfit? Dec 2 '18 at 23:05
• yea it over fits around 20 epochs. The problem in the end was just due to data samples not being enough but it think there might be some other reasons in there as well thats not immediately or obviously clear Dec 3 '18 at 9:08