0
$\begingroup$

I have a single hidden layer forward neural network build in Keras to solve a regression problem. They are time-series panel data from countries, with 2-3 countries contributing to the majority of the dataset. In total, there are around 14,000 records.

I used Keras' validation_split when training the model, which is set at 0.3, and shuffle is set to True.

It's interesting that the validation loss (mean squared error) is constantly lower than the training set, and the two losses seem to move in tandem by a constant gap. It seems like most of the time we should expect validation loss to be higher than the training loss. Is it possible that this pattern is caused by the unique composition of the validation set (i.e. turned out to be much "easier" than training set)? If that's the case, is there any way I can prove it?

P.S. I have already checked that the train/test label is correct.

enter image description here

Improve this question
$\endgroup$
1
  • $\begingroup$ Since it's time series we're dealing with, I would have tried to evaluate the model performance using increasing k-fold cross validation. Basically, you increase the training set at each step with the last block being the validation set. So if you have 4 blocks it will look like: (1,2),(12,3),(123,4) - and then you average the validation errors to get the final estimate for model performance. You may discover that this behavior you are seeing will be present only in one of the parts of the CV procedure, then you'll have more information on the nature of the problem. $\endgroup$
    – Corel
    Commented Sep 15, 2020 at 14:37

2 Answers 2

Reset to default
1
$\begingroup$

You are right to be suspicious here - I would not go on until you know what actually happens.
I'm afraid we won't be able to tell you what exactly causes this, but we may be able set you on the track of possible mechanisms that can cause such results.

One possibile explanation:

According to the documentation of Model.fit(), validation_split will reserve the last cases (rows) for the validation set. (shuffle shuffles the training data, but as I read the documentation that is after the validation set is split off).

  • If your data set exhibits some internal order, this can result in particularly easy-to-predict last cases. In other words, the validation set is not representative.

In that case, the model can still be overfitting (despite you seeing training error > validation set error). You say the data has structure, such as time series (unless all time points of a time series are the features of a single case/row) and different countries. I'd treat these factors as potentially important confounders can add correlation between rows (clusters). Until you've shown that no such clustering occurs, you basically have only two options:

  • Either decided to specify that the model should not be used outside the calibration range of e.g. the countries you have good data for, or
  • do your train/validation/test splits so that you predict unknown countries (as well as e.g. unknown time series)
Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for pointing out the caveat of validation_split. I tried 70/30 train/valid using my own partition by country (i.e. countries in each set are mutually exclusive), so that unknown countries could be predicted. I'm still observing same patterns of loss curves. One characteristic of the data is that most of the countries enter the dataset in the last 5 years (more homogeneous), with a few countries starting from 10 years ago (more variations). Could this gap just be the imbalance of the data as the testing set (30% of the countries) is less likely to include bigger variations? $\endgroup$
    – hkhk540
    Commented Aug 18, 2020 at 2:32
-1
$\begingroup$

Well, you are certainly not overfitting your model because in that case your validation error would be high, while your training error would be low. In the case of a good fit of your model the validation error should be a little bit higher than the training error in some point. There are other factors that you should consider, for example try your model with a different partition, for example of 80% for training and 10% for validation and 10% for testing. Also, maybe you are using dropout in your model, try to deactivate it in the validation phase. You can have also another possibilities, check out this link:

https://www.pyimagesearch.com/2019/10/14/why-is-my-validation-loss-lower-than-my-training-loss/

Improve this answer
$\endgroup$
1
  • $\begingroup$ While I'm not the downvoter, I so far see nothing that would reassure me here that no overfitting occurs. In particular, we don't have results for a proper model validation (in the engineering sense of validation) where we can be sure that the test data is independent. $\endgroup$
    – cbeleites
    Commented Aug 14, 2020 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.