Identifying root cause of very poor Random Forest model

Question

I've been working on building a random forest model using h2o.ai in R for climate data. I know that there is some issue, either with my understanding of randomforest, code or dataset. However, I'm not sure exactly what is causing my model to have a very high MSE and low percent variance explained. My apologies in advance if I've overlooked something very simple. I have spent much time reading and testing but haven't improved.

So far I've tried: adjusting the parameters, reducing number of correlated predictors, checking formulas and input data for outliers, normality. Based on what I've researched random forest has been used for similar data in the past. I am using 70 rows in total with a 0.7 split. The entire dataset I've created is 12M rows and to create this subset I have taken the mean for 70 regions. I have tested on the entire dataset with no significant change.

Here is my code, header and current results:

##Code##

#Check data
head(dNBR_model)
#summary(dNBR_model)

#correlation matrix
corNBR <- cor(dNBR_model)
dNBRcor <- cor.mtest(dNBR_model, conf.level = 0.95)

corrplot(corNBR, p.mat = dNBRcor$p, type = "upper", order = "hclust", insig='blank', addCoef.col ='black', tl.col = "black", tl.srt = 45)

#Run RF model
set.seed(561)
dNBR_split <- initial_split(dNBR_model, prop = .7)
dNBR_train <- training(dNBR_split)
dNBR_test <- testing(dNBR_split)

y <- "dNBR"
x <- setdiff(names(dNBR_train), y)

#initialize h2o
h2o.init(max_mem_size='50G')

#convert train to h2o
train.h2o <- as.h2o(dNBR_train)
dNBR_test.h2o <- as.h2o(dNBR_test)

testDRF <- h2o.randomForest(x, y, ntrees = 500, max_depth = 15, min_rows = 1, mtries = 7, nbins = 20, sample_rate = 0.75000, training_frame = train.h2o, validation_frame = dNBR_test.h2o)

testperf <- h2o.performance(testDRF)
summary(testDRF)

#percent variance explained
VE = ((1 - h2o.mse(testDRF))/(h2o.var(train.h2o$dNBR)))*100
print(VE)

RMSE = h2o.mse(testDRF) %>% sqrt()
PRMSE = (RMSE/(mean(dNBR_test$dNBR)))*100 
print(PRMSE)

h2o.varimp_plot(testDRF)
varimp <- h2o.varimp(testDRF)

h2o.residual_analysis_plot(model = testDRF, newdata = dNBR_test.h2o)

##Results##

** Reported on validation data. **

MSE:  4034.157
RMSE:  63.51502
MAE:  47.38234
RMSLE:  0.1174528
Mean Residual Deviance :  4034.157

% variance explained: -167.545

##Header##

Answer 1

(In the comments, I see you think it solved, so this is more general advice.)

To identify the root cause of a poor model, you should start by getting a baseline model to compare against.

As you are using H2O, that is very easy, as you can just swap h2o.randomForest with, say, h2o.glm (for generalized linear model). You could also try h2o.deeplearning for a neural net, but using a linear model as a baseline is often good as it is quick, and the defaults are good.

So if the results are equally bad, the data or how it is loaded becomes the suspect. If the glm is much better, the way you are using randomForest becomes suspect.

With H2O, you have H2O Flow: open a web browser at http://localhost:54321/ and you can see the models and data you have loaded in, and you can then explore and analyze it.

Another thing, which also came up in the comments, is that if you are overfitting you should be seeing good scores when evaluating on your training data. If you don't then I'd be suspicious of the data again.

E.g. with this data, the very best training model score you could hope for is 0.33.

x1,x2,y
1,2,A
2,3,A
1,2,B
2,3,B
1,2,C
2,3,C

This kind of thing can happen if the data has been loaded badly, e.g. it thought the data was csv, but it was actually semi-colon separated. Again, viewing the data in H2O Flow can confirm if this has happened.