I've run a RandomForestRegressor (Scikit Ensemble) over N loops, each time changing the random seed and therefore changing the train test split. This way I've N sets of predictions (M predictions for each loop). I've also captured the following data
R2_Score for Train Dataset over N Loops (1 value per loop)
R2_Score for Test Dataset over N Loops (1 value per loop)
Currently, I average out all the predictions for which R2_Score_Train > Median(R2_Score_Train) && R2_Score_Test > Median(R2_Score_Test)
I want to know a better way to make out the final predictions, using the R2_Score for both train and test dataset. One way I was thinking to is to give more weight to a loop for which R2_Score_Train ~ R2_Score_Test (i.e. the difference between them is the smallest) and lowering the weights for which this value is high.
I am using Python for this. Scikit-Learn. Wanted to know if there is any inbuilt function or any 3rd party library?