3
$\begingroup$

I am performing a regression on a Dataset and try to replace a mathematical Model with a Neural Network.

To avoid overfitting I decided to use the Early Stopping Callback Function of Keras. So far I have been told, that the metrics to monitor for this problem should be "val_loss" but when I try to do that, the neural networks stops training very early.

I tried to monitor the mean absolute Error 'mae' and seem to get much better results, but I found no other example doing this so I am not sure if I am making another mistake which I am not aware of. Is there a reason not to do this? Or is there maybe even another metric that i should consider using?

dataset = np.loadtxt("output_20180804.out", delimiter=",")

X  = dataset[0:5000,4:7]
Y  = dataset[0:5000,0:4]

tbCallBack = TensorBoard(log_dir='./Graph{}', histogram_freq=0, write_graph=True, write_images=True) #TensorBoard Monitoring
esCallback = EarlyStopping(monitor='val_loss',
                           min_delta=0,
                           patience=2,
                           verbose=1,
                           mode='auto')


#create Layers
visible = Input(shape=(3,))
x = Dropout(.1)(visible)
x = Dense(60)(x)
output = Dense(4)(x)


Optimizer = optimizers.Adam(lr=0.0001
                            #amsgrad = True
                           )
model = Model(inputs=visible, outputs = output)
model.compile(optimizer=Optimizer,
              loss=['mse'],
              metrics=['mae']
              )
model.fit(X, Y, epochs=100000, batch_size=200, shuffle=True, validation_split=0.35, callbacks=[tbCallBack, esCallback])




print(model.get_weights())



# evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))


savestring = 'dense_100_epo100000_batch800_val02_20180807.h5'
model.save(savestring)

model.summary()
$\endgroup$
1
  • $\begingroup$ val_loss is just the validation set loss, mae can be used to measure it. What exactly is your question? Do you ask if MAE can be used as a loss function? Sure it can. If you optimize for MAE, then it is perfectly reasonable to use it in early stopping. $\endgroup$
    – Tim
    Aug 13 '18 at 12:23
5
$\begingroup$

In order to prevent overfitting, EarlyStopping should monitor a validation metric. Because your loss function is the mse, by monitoring val_loss you are essentially monitoring the validation Mean Squared Error. If you think that mae is a better metric for your task, you should monitor val_mae instead.


Why monitor a validation metric when performing early stopping?

Early stopping, is mostly intended to combat overfitting in your model. Overfitting is a phenomenon, commonly occurring in Machine Learning, where a model performs worse on a validation/test set than the training set.

Take a look at the image below:

The red and blue lines depict the model's loss, during training, on the training/validation sets respectively. As you can see, while the training loss continues to drop, the validation loss initially plateaus and then starts to raise. This happens because after some point the model learns to memorize the training data, while forgetting how to effectively differentiate between the classes. When we reach this point, even though we think the model is improving (indicated by all the training set metrics), actually has begun to diverge from its goal (to perform well on unseen data).

Early stopping is a method of combating this. By terminating the model, before it has completed its training we might get a better performance on unseen data. This works by monitoring a validation metric and terminating the model when this metric stops dropping.

$\endgroup$
2
  • $\begingroup$ You are saying "validation metric" when you mean validation loss. This can be confusing because the (performance) metric is not the same as the loss. In other words, Early Stopping is typically used on the validation loss, not the validation metric. That said, sometimes people do use the validation metric instead. $\endgroup$ Oct 21 at 23:11
  • $\begingroup$ I purposefully said "validation metric" to keep it more general. You could theoretically stop training early based on any performance metric you monitor on the validation set, during training. $\endgroup$
    – Djib2011
    Oct 23 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.