How to evaluate the quality of predictions? Say I have a big sample of data (simple positive numbers), and I want to check my prediction quality by calibrating my "model" (predictor?) based on the initial part of the sample, generating a prediction and comparing it to the real-life outcome. I have (and understand) the measures of the quality of predictions, but I am missing the intervals design. 
Say, the data is $[a_0, a_1, a_2, ..., a_k, ..., a_z]$. I would take $[a_0, a_1, a_2, ..., a_k]$ as the known data, generate $[a_{k+1}, ... a_z]$ and compare the quality. How to choose the size of "given" sample and the size of the sample I am trying to predict? 
Does it make sense to make multiple predictions of smaller intervals, e.g. predict $[a_{k+1}, ..., a_l]$ using $[a_0, a_1, a_2, ..., a_k]$, then predict $[a_{l+1} ..., a_m]$ using $[a_0, a_1, a_2, ..., a_k ..., a_l]$, etc.? Again, how to design the sizes of the samples?
 A: Typically, how far out you predict (often called the "forecasting horizon" or similar) is governed by what you will use the forecasts or predictions for.
For instance, I do forecasting for supermarket replenishment. A supermarket may order for a given assortment once a week, e.g., on Tuesday evening, and the order will arrive and be on the shelves by Thursday morning. Therefore, my forecast today (on Tuesday) needs to cover the next eight days, until next Wednesday, because that is the time period that today's order needs to cover - after that, we will have new stock, ordered one week later.
You will very rarely have the luxury of setting your forecasting horizon yourself.
Of course, you may find out that you can't do a very good job at predicting more than a certain period out. For instance, weather forecasts are only better than chance for maybe ten or fourteen days out. Anything beyond that may not be amenable to a more sophisticated model than using the long-term average. Which is just another way of saying that you need to provide predictions for whatever length your forecast consumer requires - but you might use different models for different horizons.
