I have a dataset with 46 million observations and 25 predictors. I am training my model in the python h2o package like so:
import h2o h2o.init(nthreads = -1, max_mem_size = "110g") df = h2o.import_file(path = "C:/user/path_to_data.csv") #predictors x = [col for col in df.column if 'target' not in col] y = 'target' #establish model m = h2o.estimators.H2ORandomForestEstimator(model_id="RF_defaults", nfolds = 5) #train m.train(x,y, df) #view results print (m.cross_validation_metrics_summary())
My result shows r2 values of about 0.93 across all 5 folds.
There is one variable in particular I am interested in seeing how predicted values compare to, called
Year Since Fire. To examine this I randomly selected 1,000 observations that were trained in the model, and then replicated each 70 while changing year of fire so that each ranged from 1 to 70, and kept all other variables the same as what they originally were. In this way I then had 70,000 observations.
I then used the model to predict (
m.predict(new_data))the target variable through all 70,000 of these observations. I then plotted the mean predictions for each Year Since Fire, and compare to the mean of the true observations in the original 1,000 pixels.
This is what the plot looks like (Albedo is the target variable):
and the r2 of this is only 0.13.
My main question is, if my cross validated r2 values or so high, why is the model predicting so poorly when I do this simulation exercise?
My initial thought was the variable
Year Since Fire was not capturing most of the variation (it is number 5 in feature importance and explains 0.08 percent of the variation). To test this I built 70 different models, one for each Year Since Fire and then applied the model appropriately, e.g. all pixels I wanted to predict on in Year Since Fire 1 were only modeled with a model built on Year Since Fire 1. This is result is better, but still not great with an r2 of 0.6, and I am sure there is error being propagated by building 70 different models in the first place.
To provide information on how the data is sampled and simulated, 1 of the 1,000 sample pixels may look like this (without showing all 25 predictors):
Albedo Year_Since_Fire Temp Prcp DD 0.6 50 20 100 300
This 1 observation is then replicated 70 times, while changing Year_Since_Fire:
Year_Since_Fire Temp Prcp DD 1 20 100 300 2 20 100 300 3 20 100 300 4 20 100 300 5 20 100 300 6 20 100 300 ................................. 70 20 100 300
This done for all 1,000 pixels, each with different initial values for the independent variables.