2
$\begingroup$

Say I have a big sample of data (simple positive numbers), and I want to check my prediction quality by calibrating my "model" (predictor?) based on the initial part of the sample, generating a prediction and comparing it to the real-life outcome. I have (and understand) the measures of the quality of predictions, but I am missing the intervals design.

Say, the data is $[a_0, a_1, a_2, ..., a_k, ..., a_z]$. I would take $[a_0, a_1, a_2, ..., a_k]$ as the known data, generate $[a_{k+1}, ... a_z]$ and compare the quality. How to choose the size of "given" sample and the size of the sample I am trying to predict?

Does it make sense to make multiple predictions of smaller intervals, e.g. predict $[a_{k+1}, ..., a_l]$ using $[a_0, a_1, a_2, ..., a_k]$, then predict $[a_{l+1} ..., a_m]$ using $[a_0, a_1, a_2, ..., a_k ..., a_l]$, etc.? Again, how to design the sizes of the samples?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I took the liberty of removing the prediction-interval tag. PIs refer to quantile forecasts, e.g. forecasting two numbers for a given future time period such that we hope that the actual outcome lies between the two numbers with 95% confidence or similar. This is necessary in safety stock calculation. But it's a different thing than what you are asking about. $\endgroup$
    – Stephan Kolassa
    Commented Feb 25, 2020 at 16:34

1 Answer 1

Reset to default
2
$\begingroup$

Typically, how far out you predict (often called the "forecasting horizon" or similar) is governed by what you will use the forecasts or predictions for.

For instance, I do forecasting for supermarket replenishment. A supermarket may order for a given assortment once a week, e.g., on Tuesday evening, and the order will arrive and be on the shelves by Thursday morning. Therefore, my forecast today (on Tuesday) needs to cover the next eight days, until next Wednesday, because that is the time period that today's order needs to cover - after that, we will have new stock, ordered one week later.

You will very rarely have the luxury of setting your forecasting horizon yourself.

Of course, you may find out that you can't do a very good job at predicting more than a certain period out. For instance, weather forecasts are only better than chance for maybe ten or fourteen days out. Anything beyond that may not be amenable to a more sophisticated model than using the long-term average. Which is just another way of saying that you need to provide predictions for whatever length your forecast consumer requires - but you might use different models for different horizons.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The task is to compare several models on many large sets of data; I am wondering what is the best way to do this. This is not necessarily the most sensible exercise, but I still want to know how to solve it in the best possible way, if it can make sense at all :) $\endgroup$
    – Yulia V
    Commented Feb 25, 2020 at 16:34
  • 1
    $\begingroup$ In that case, I would try to figure out what horizon is most relevant and compare the models on that specific horizon. After all, different models may be optimal for different horizons. There is no "overall best model" divorced from the use you want to put your forecasts to. $\endgroup$
    – Stephan Kolassa
    Commented Feb 25, 2020 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.