I am creating a series of monthly models to predict the yearly production of grain for a fixed US state. For a given month, the model is built using the data from the past 10 years and it allows me to predict the target value of this year. Since the target is a yearly value, for a given year it is the same for each monthly model. My idea is that as we get closer to the end of the year, we have more data and the prediction should get better.

Each monthly model has features based on intrinsic data (such as latitude and longitude), data from the previous year (for example population, tons of fertilizer used, etc), and data from previous months of the same year (so if I consider the March model, it will contain the average temperature of January and of February). The dataset of month M will have more features than the dataset of month M-1, since it includes all the features of month M-1 and the "extra" information which I know only after month M-1 has terminated (for example, the February dataset will contain the average temperature of January and not the average temperature of February. The March dataset will contain the average temperature of February). The logic behind this is that in month M I know everything I knew in month M-1 and something more! I first concentrated on obtaining and tuning the monthly models. For each month I considered a class of models (for example linear regression, random forest, etc) and obtained the "best" model by tuning the hyperparameters using a moving window. For simplicity let's consider a random forest. Once I obtained the 12 random forests, I expected to find that the prediction of month M was better than the prediction of month M-1. However, what I instead obtained was that for a fixed year the performance of the models got worse as months passed. If I plot the RMSE I get a growing trend through the months!

My question is why? Month M has all the data of month M-1, so why should it perform worse?

ps- I also want to note that I did some tests. I initially thought this could be due to the fact that the number of features grows while the number of observations is constant, however if I use feature selection to reduce all the monthly datasets to the same number of features I still get the same trend of the RMSE.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.