Understanding train vs validation loss chart

Question

I am training an LSTM to a univariate time series and I have some questions about how to evaluate the train vs validations loss charts and which number of epochs to use in the model.

To give more context about my data. It is a monthly univariate time series and the LSTM wants to predict the next 12 data points. The data is in sliding window format with 12 inputs and 12 outputs. A summary of the model is below.

1. In both charts I see that the error in the validation dataset is smaller than the error in the training set. It means that I cannot generalize well so I am underfitting, right? The training and validation loss seems to converge around 40 epochs for the MAE loss and for the MSE.

2. Should I use MAE as loss? As far as I know, MAE and MSE are the error metrics generally used for time series.

3. Which number of epochs should I use for this model?

#DEFINE THE MODEL
lstm_model <- keras_model_sequential()
  lstm_model %>%
  layer_lstm(units = 12, #24, # size of the layer
       batch_input_shape = c(1, 12, 1), # batch size, timesteps, features
       return_sequences = TRUE,
       stateful = TRUE,
       name = "LSTM") %>%
  time_distributed(keras::layer_dense(units = 1), name = "Output")

  #COMPILE
    lstm_model %>%
    compile(loss = 'mae', optimizer = optimizer_adam(lr = 0.001, decay = 1e-6), metrics = 'mse')
  summary(lstm_model)
 
 #FIT THE MODEL
  validation_split = 0.25 
    train_history = lstm_model %>% 
    fit(
    x = x_train_arr,
    y = y_train_arr,
    batch_size = 1,
    epochs = 100,
    verbose = 1,
    validation_split = validation_split,
    shuffle = FALSE
    )

Answer 1

To do early stopping, you need to find the minimum of the validation loss MSE curve which happens to be around the 26th epoch. After this, the validation loss increases and thus your model starts to overfit to the training data.

1. When the validation loss is below the training loss that means your model is already genralizing well. If you intend to reduce the training loss further, you can increase the complexity of your model by increasing number of layers and/or neuron size. To reduce validation loss, you need to reduce the complexity by regularization using dropout, so you need to do some tradeoff. Another way to reduce the validation loss is to make the training set more representative of the validation/test set. Increase number of training samples in each epoch by adding new training samples to the training dataset. Make sure it captures all the variations.

2. I would use MSE/MAE instead of the loss because eventually you will use MSE/MAE to evaluate your test set and MSE/MAE score on validation set is a good representation of the anticipated MAE in test set.

3. use 25 or 26 epochs. Remember to use the validation set in your training after choosing the hyperparameters. This helps to increase the training data size slightly.