More Statistical Way to Average N Predictions

Question

I've run a RandomForestRegressor (Scikit Ensemble) over N loops, each time changing the random seed and therefore changing the train test split. This way I've N sets of predictions (M predictions for each loop). I've also captured the following data

R2_Score for Train Dataset over N Loops (1 value per loop)

R2_Score for Test Dataset over N Loops (1 value per loop)

Currently, I average out all the predictions for which R2_Score_Train > Median(R2_Score_Train) && R2_Score_Test > Median(R2_Score_Test)

I want to know a better way to make out the final predictions, using the R2_Score for both train and test dataset. One way I was thinking to is to give more weight to a loop for which R2_Score_Train ~ R2_Score_Test (i.e. the difference between them is the smallest) and lowering the weights for which this value is high.

I am using Python for this. Scikit-Learn. Wanted to know if there is any inbuilt function or any 3rd party library?

Answer 1

Generally an average of a set of forecasts gives a better forecast than any individual forecast, so why not use all the forecasts, even the poor ones? The question - you also ask - is how to do this. The most simple way is to use weighted mean, where you weight each forecast by the inverse error. This makes 'poor' forecasts have a low weight and the 'good' ones to have high weight. There are more sophisticated approaches, check the references for learning more.

---

Winkler and Makridakis (1983). The Combination of Forecasts. J. R. Statis. Soc. A. 146(2), 150-157.

Makridakis and Winkler (1983). Averages of Forecasts: Some Empirical Results. Management Science, 29(9) 987-996.

Clemen (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559-583.

Bates, J. M., & Granger, C. W. (1969). The combination of forecasts. Or, 451-468.

Answer 2

There are "more statistical" ways, but they're not better. It's known as a "forecast combination puzzle", when a simple average outperforms other more sophisticated weighting schemes. I don't think there's a definitive answer to why it happens but you can find a lot of papers on this subject, e.g. "A Simple Explanation of the Forecast Combination Puzzle" Jeremy Smith, Kenneth F. Wallis

My explanation is that your weighting schemes are based on the estimates themselves. You don't know the optimal weights, you must estimate them. This brings in the estimation error, which when combined with the individual forecast errors in aggregation increases the combined forecast error.