0
$\begingroup$

I have a time series dataset that covers daily observations (closing price) for several stocks, and I would like to build models to forecast the closing prices for the future 7 days using their historical closing prices (up to 21 days) as the features.

I splitted the data into training (first 9 months with shuffling) and validation set (last 3 months without shuffling) and built a MLP model on the training set. However, it is surprising that the training error (either MSE or MAE) is always significantly higher than the validation set. This is also the case for very naive model such as moving average. Is this normal in time-series forecast? What could be the reasons? Thanks!

Edit:

If I randomly split the data into 80% training and 20% validation, the errors are comparable, with validation error slightly higher, which makes the above issue even stranger. Why would different splitting methods yeild to different error patterns?

$\endgroup$
  • $\begingroup$ Have you tried using another period (like the first 3 months) as a test set? This way you could be able to tell if the problem is in the data (the last 3 months could be more regular than previous ones) or in your analysis (you may find the same problem with every test set). $\endgroup$ – Pere Jun 10 at 18:27
  • $\begingroup$ I did a quick check on the first three months using moving average. Indeed the error on the first 3 monhts are much higher than the error on the last three months. I wonder if this suggests what you described as "last 3 months could be more regular"? What metrics could better reveal this (std? mean?)? Thanks! $\endgroup$ – Lynn Jun 10 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.