# Random Forest cross-validation r2 is high but predictions on simulated data are bad

I have a dataset with 46 million observations and 25 predictors. I am training my model in the python h2o package like so:

import h2o

h2o.init(nthreads = -1, max_mem_size = "110g")
df = h2o.import_file(path = "C:/user/path_to_data.csv")

#predictors
x = [col for col in df.column if 'target' not in col]
y = 'target'

#establish model
m = h2o.estimators.H2ORandomForestEstimator(model_id="RF_defaults", nfolds = 5)

#train
m.train(x,y, df)

#view results
print (m.cross_validation_metrics_summary())


My result shows r2 values of about 0.93 across all 5 folds.

There is one variable in particular I am interested in seeing how predicted values compare to, called Year Since Fire. To examine this I randomly selected 1,000 observations that were trained in the model, and then replicated each 70 while changing year of fire so that each ranged from 1 to 70, and kept all other variables the same as what they originally were. In this way I then had 70,000 observations.

I then used the model to predict (m.predict(new_data))the target variable through all 70,000 of these observations. I then plotted the mean predictions for each Year Since Fire, and compare to the mean of the true observations in the original 1,000 pixels.

This is what the plot looks like (Albedo is the target variable):

and the r2 of this is only 0.13.

My main question is, if my cross validated r2 values or so high, why is the model predicting so poorly when I do this simulation exercise?

My initial thought was the variable Year Since Fire was not capturing most of the variation (it is number 5 in feature importance and explains 0.08 percent of the variation). To test this I built 70 different models, one for each Year Since Fire and then applied the model appropriately, e.g. all pixels I wanted to predict on in Year Since Fire 1 were only modeled with a model built on Year Since Fire 1. This is result is better, but still not great with an r2 of 0.6, and I am sure there is error being propagated by building 70 different models in the first place.

EDIT:

To provide information on how the data is sampled and simulated, 1 of the 1,000 sample pixels may look like this (without showing all 25 predictors):

Albedo  Year_Since_Fire  Temp  Prcp  DD
0.6     50               20    100   300


This 1 observation is then replicated 70 times, while changing Year_Since_Fire:

 Year_Since_Fire  Temp  Prcp  DD
1                20    100   300
2                20    100   300
3                20    100   300
4                20    100   300
5                20    100   300
6                20    100   300
.................................
70               20    100   300


This done for all 1,000 pixels, each with different initial values for the independent variables.

• Do sample units appear in the data more than once? For example, is a single unit replicated in many different rows, each showing year 0 through 70 with results in each line? – Underminer Dec 18 '17 at 20:49
• @Underminer If I understood your question correctly, then yes. I provided more information in my question. – Stefano Potter Dec 18 '17 at 21:07
• The reason for your miss is likely caused by overfitting caused by auto correlation of sampling units. The model sees 20, 100, 300 as predictors, and thinks it "knows" what the Albedo will be (because it has seen other similar samples from the same unit). The differences between units is describing the variability better than the Year_Since_Fire so it is leaning on that data more. A model that takes into account the auto correlation should fix this. – Underminer Dec 18 '17 at 21:22
• How are you choosing cross validation sets? Randomly selecting folds? – Underminer Dec 18 '17 at 21:24
• Yes, I believe this is also why 70 different models works better. Could you provide a quick example of where to start to learn a model with auto correlation? I know the concept but have not applied it. For the folds yes they are randomly chosen in the python package, and I have also done a 80% train and 20% test with similar results to the folds, with the 80 and 20 completely random. – Stefano Potter Dec 18 '17 at 21:28

For the question in your subject, I think the answer is that r2 is poor metric to be using. See http://data.library.virginia.edu/is-r-squared-useless/

I would instead use RMSE to judge the quality of your models. (Just preferring RMSE over MSE as it is in actual units, so easier to judge if "4.5" is a good enough model or not.) (I'd also be looking at MAE; if RMSE is much worse than MAE you could describe it as "When it gets it wrong, it really gets it wrong.")

Also, just to be aware, though I am sure you have enough data that this won't be a factor: the final model is built using all your data, not the 80% that each CV model uses. It is worth comparing the RMSE of each of the 5 CV models with the RMSE of the final model. If they are all close, then the random fold selection is reliable; if they are wildly different, you may need to try a different way of making the folds.

As for the rest of the question:

To examine this I randomly selected 1,000 observations that were trained in the model, and then replicated each 70 (times) while changing year of fire so that each ranged from 1 to 70, and kept all other variables the same as what they originally were

Unless I have have misunderstood, the behaviour you see makes sense for a tree algorithm, i.e. a non-linear algorithm. The "all the other variables" will send it down certain parts of the tree before it considers Year. If the falsified value for Year is something it has not seen before, it will be choosing the value nearest to what it has seen.

As an extreme example, if you have trained with:

a   b    answer
3   5    8
2   3    5


Then you change the b to be all 3s, it will give this result if a tends to come before b in the trees:

a   b    prediction
3   3    8
2   3    5


And this result if b tends to come first:

a   b    prediction
3   3    5
2   3    5

• This makes sense, Year is just not being used to split the trees very high up. I also use RMSE to validate and it is stable, but wasn't aware of the RMSE MAE relation, I will look. P.S. I really enjoy the practical machine learning h2o book. – Stefano Potter Dec 19 '17 at 14:12