# How to evaluate the quality of predictions?

Say I have a big sample of data (simple positive numbers), and I want to check my prediction quality by calibrating my "model" (predictor?) based on the initial part of the sample, generating a prediction and comparing it to the real-life outcome. I have (and understand) the measures of the quality of predictions, but I am missing the intervals design.

Say, the data is $$[a_0, a_1, a_2, ..., a_k, ..., a_z]$$. I would take $$[a_0, a_1, a_2, ..., a_k]$$ as the known data, generate $$[a_{k+1}, ... a_z]$$ and compare the quality. How to choose the size of "given" sample and the size of the sample I am trying to predict?

Does it make sense to make multiple predictions of smaller intervals, e.g. predict $$[a_{k+1}, ..., a_l]$$ using $$[a_0, a_1, a_2, ..., a_k]$$, then predict $$[a_{l+1} ..., a_m]$$ using $$[a_0, a_1, a_2, ..., a_k ..., a_l]$$, etc.? Again, how to design the sizes of the samples?

• I took the liberty of removing the prediction-interval tag. PIs refer to quantile forecasts, e.g. forecasting two numbers for a given future time period such that we hope that the actual outcome lies between the two numbers with 95% confidence or similar. This is necessary in safety stock calculation. But it's a different thing than what you are asking about. – Stephan Kolassa Feb 25 '20 at 16:34