8
$\begingroup$

I would like to train a simple neural network to forecast electricity prices in a certain region. However, I only have a 'limited' amount of data available (3 sequential years of the historical price, electricity generation and load in that region with 1 hour intervals – no other data import allowed). First of all, I'm aware that it's not allowed to split up the data randomly because it's a time series. Next, Literature recommends splitting up in a 60-20-20, 70-15-15 or 80-10-10 fashion to obtain the train, validation and test set. The split "2 years, 0.5 years, and 0.5 years" for train, validation, and test set respectively would result in hyperparameter optimization for the period January to June and testing the model for July to December. Intuitively this doesn't feel like a good approach to me because this would cause a mismatch in seasonality in the validation and test set. Is this intuition correct and if so, what alternative approach could be taken?

$\endgroup$

1 Answer 1

8
$\begingroup$

The seasonality would be different between the validation and the test set. I would not call it a mismatch.

Indeed, this is a very common setup in time series forecasting, and yes, any results need to be treated with caution because you would be evaluating not over a full (yearly) seasonal cycle. But with the amount of data you have, there is quite simply little else you can do. More data is always helpful, and not always available.

One other way to evaluate time series forecasts is rolling origin evaluation (also called time series cross validation), but that quite simply won't address the issue present here.

You could complement your analysis by a "backward evaluation": turn your time series around in time and fit & forecast backward. Then if your original evaluation used the last six months of data (July to December), then this "backward evaluation" would use the first six months (January to June). I would not necessary trust this more than the original evaluation, but it might be enlightening.

Take a specific model (e.g., ETS(A,N,N)), fit to 1/2021 to 12/2022, test on 1-6/2023, refit and validate on 7-12/2023. Then take the same model and let time run backwards, fitting to 12/2023 to 1/2022, test on 12-7/2021, refit and validate on 6-1/2021. Now you have two evaluations of your model, one on July to December and one on January to June. Of course the two evaluations will be on same model fitted with different parameters - but it tells you something about the model form as such, and in production, you would refit it anyway.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. Could you specify in what sense the proposed "backward evaluation" could yield enlightening results? $\endgroup$
    – Robbe
    Apr 24 at 12:15
  • $\begingroup$ I edited my answer. Essentially, it gives you an evaluation on the part of the yearly seasonal cycle that you didn't see the first time around. Again, this should be taken with a grain of salt, but it might yield useful results. $\endgroup$ Apr 24 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.