What degree of difference does validation and training loss need to have to be called good fit?

Question

I am conducting a multi-variate time series forecasting using an LSTM model. The model architecture and other details are given below: Dataset split: (80/10/10 split)

Training Data Points: 367640

Validation Data Points: 45388

Test Data Points: 40849

Features Scaling: (After data splitting)

MinMax Scaler(-1,1)

Model Architecture:

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 lstm (LSTM)                 (None, 24, 256)           272384    
                                                                 
 lstm_1 (LSTM)               (None, 100)               142800    
                                                                 
 dense (Dense)               (None, 1)                 101       
                                                                 
=================================================================
Total params: 415,285
Trainable params: 415,285
Non-trainable params: 0
_________________________________________________________________

Activation Function in hidden layers: tanh Activation Function in output layer: None

Model & Training Parameters:

Learning rate: 0.0001 Batch Size: 128 Epochs: 25

****Loss Curve ****

As can be observed from the training and validation loss curve, the validation loss becomes less than the training loss after 9th epoch. From what I learned from online resources:

• The model fit where the validation loss is less than the training loss represnts an unkown fit.
• One intuitive reasoning in this case with absence of any kind of regularization is validation set examples are relatively easier to learn.
• What I have tried is changing batch size and learning rate such that increasing learning rate and learning rate reduces the loss difference although very minimally(BS:128-->256, LR: 0.0001-->0.001) results in validation loss to 0.0006 from 0.0007.
• Another finding is that increasing the number of hidden layers(2-->3, almost similar model parameters) tend to reduce the loss difference to about 0.0002.

The main issue in each of the above mentioned cases is that the validation error is less than training error, So my questions are:

1. Is it acceptable that validation loss curve is lower than the training loss curve GIVEN THAT DIFFERENCE BETWEEN THE TWO IS VERY SMALL?
2. Can this trained model be categorized as a good model?
3. Generally, how much difference between the validation curve and training curve is acceptable as in almost all of the cases I have found online this difference is zero?
4. What are the potential solutions to overcome this problem ?

Answer 1

It's hard to tell without understanding the data or having a baseline model to compare your model to it

Without Knowing It might be several things:

• Validation Data Might be Easier as you said
• The dataset is easily predicted overall (for example by taking the average of the last 10 days I can predict Y)

Now to answer your questions:

1. Yes It's acceptable in that range, although if it's diverging or the difference is high it might be that the validation data is straightforward and small

2. This is a tricky question to answer because it depends on the use case and the metric you're using Judging from the loss only isn't going to help you, you need to check, If it's a regression problem check what's the average error you'd accept in this case and so on or compare to a baseline model.

3. Sadly you can't add a number, and you need a metric specific to your use case to decide, but generally, you want the difference to be constant across the training epochs(Both the training and validation loss are moving together).

4. Define a good metric to decide the performance of your model, use your test data to evaluate your model, and compare your model to available benchmarks or baseline