I am conducting a multi-variate time series forecasting using an LSTM model. The model architecture and other details are given below: Dataset split: (80/10/10 split)
Training Data Points: 367640
Validation Data Points: 45388
Test Data Points: 40849
Features Scaling: (After data splitting)
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= lstm (LSTM) (None, 24, 256) 272384 lstm_1 (LSTM) (None, 100) 142800 dense (Dense) (None, 1) 101 ================================================================= Total params: 415,285 Trainable params: 415,285 Non-trainable params: 0 _________________________________________________________________
Activation Function in hidden layers: tanh Activation Function in output layer: None
Model & Training Parameters:
Learning rate: 0.0001 Batch Size: 128 Epochs: 25
****Loss Curve ****
As can be observed from the training and validation loss curve, the validation loss becomes less than the training loss after 9th epoch. From what I learned from online resources:
- The model fit where the validation loss is less than the training loss represnts an unkown fit.
- One intuitive reasoning in this case with absence of any kind of regularization is validation set examples are relatively easier to learn.
- What I have tried is changing batch size and learning rate such that increasing learning rate and learning rate reduces the loss difference although very minimally(BS:128-->256, LR: 0.0001-->0.001) results in validation loss to 0.0006 from 0.0007.
- Another finding is that increasing the number of hidden layers(2-->3, almost similar model parameters) tend to reduce the loss difference to about 0.0002.
The main issue in each of the above mentioned cases is that the validation error is less than training error, So my questions are:
- Is it acceptable that validation loss curve is lower than the training loss curve GIVEN THAT DIFFERENCE BETWEEN THE TWO IS VERY SMALL?
- Can this trained model be categorized as a good model?
- Generally, how much difference between the validation curve and training curve is acceptable as in almost all of the cases I have found online this difference is zero?
- What are the potential solutions to overcome this problem ?