I try to compare the forecasting performance of several models. I do it for two situations: normal and extreme cases. My dataset set is not big. One of the models (gradient boosting) requiers a minimal training set which is bigger than the other models (45), which makes the extreme cases check a bit useless. In each iteration, I add another observation, so in total I get X predictions.
My question: would it be acceptable to use a smaller first training set (30, for example) for all the non-gradient boosting models, so to have more extreme cases to examine? I suspect it would make the comparision statistically biased. On the other hand, it would improve my forecasting ability.