1
$\begingroup$

I've been working on building a random forest model using h2o.ai in R for climate data. I know that there is some issue, either with my understanding of randomforest, code or dataset. However, I'm not sure exactly what is causing my model to have a very high MSE and low percent variance explained. My apologies in advance if I've overlooked something very simple. I have spent much time reading and testing but haven't improved.

So far I've tried: adjusting the parameters, reducing number of correlated predictors, checking formulas and input data for outliers, normality. Based on what I've researched random forest has been used for similar data in the past. I am using 70 rows in total with a 0.7 split. The entire dataset I've created is 12M rows and to create this subset I have taken the mean for 70 regions. I have tested on the entire dataset with no significant change.

Here is my code, header and current results:

##Code##

#Check data
head(dNBR_model)
#summary(dNBR_model)

#correlation matrix
corNBR <- cor(dNBR_model)
dNBRcor <- cor.mtest(dNBR_model, conf.level = 0.95)

corrplot(corNBR, p.mat = dNBRcor$p, type = "upper", order = "hclust", insig='blank', addCoef.col ='black', tl.col = "black", tl.srt = 45)

#Run RF model
set.seed(561)
dNBR_split <- initial_split(dNBR_model, prop = .7)
dNBR_train <- training(dNBR_split)
dNBR_test <- testing(dNBR_split)

y <- "dNBR"
x <- setdiff(names(dNBR_train), y)

#initialize h2o
h2o.init(max_mem_size='50G')

#convert train to h2o
train.h2o <- as.h2o(dNBR_train)
dNBR_test.h2o <- as.h2o(dNBR_test)

testDRF <- h2o.randomForest(x, y, ntrees = 500, max_depth = 15, min_rows = 1, mtries = 7, nbins = 20, sample_rate = 0.75000, training_frame = train.h2o, validation_frame = dNBR_test.h2o)

testperf <- h2o.performance(testDRF)
summary(testDRF)

#percent variance explained
VE = ((1 - h2o.mse(testDRF))/(h2o.var(train.h2o$dNBR)))*100
print(VE)

RMSE = h2o.mse(testDRF) %>% sqrt()
PRMSE = (RMSE/(mean(dNBR_test$dNBR)))*100 
print(PRMSE)

h2o.varimp_plot(testDRF)
varimp <- h2o.varimp(testDRF)

h2o.residual_analysis_plot(model = testDRF, newdata = dNBR_test.h2o)

##Results##

** Reported on validation data. **

MSE:  4034.157
RMSE:  63.51502
MAE:  47.38234
RMSLE:  0.1174528
Mean Residual Deviance :  4034.157

% variance explained: -167.545

residuals plot

##Header## header for input data

$\endgroup$
13
  • 1
    $\begingroup$ Welcome to Cross Validated! 1) How do you calculate the percent of variance explained? Do you know the math or at least the software command? 2) When you say that you have 70 rows, do you mean that you have 70 observations in the model, from which you allocation 70% to training and 30% to testing? When you write that you "have tested on the entire dataset with no significant change," do you mean that you have applied your pipeline to all 12-million observations? $\endgroup$
    – Dave
    Commented Mar 16, 2023 at 0:17
  • $\begingroup$ To clarify 1) I've calculated percent variance explained as 1 - MSE/var(y) * 100 I couldn't find a command to do this in h2o so I had to write it. 3) Yes, this model has 70 observations with 70% training, 30% testing. Yes, I also tried with all 12-million observations. $\endgroup$ Commented Mar 16, 2023 at 0:21
  • $\begingroup$ Next, how is your in-sample performance? // If 1-MSE/var(y) < 0, do you see what that says about MSE/var(y)? $\endgroup$
    – Dave
    Commented Mar 16, 2023 at 0:23
  • 2
    $\begingroup$ Overfitting is a thought, but the fact that you have such poor performance even in-sample suggests another culprit. Overfitting leads to poor out-of-sample performance, but kind of the whole point is that overfitting leads to an in-sample performance that does not generalize, yet your in-sample performance is worse than that of a model that predicts $\bar y$ every time, as I discuss here. This is hardly the sign of a model that has detected coincidences in the data and fit to them; the model cannot even do that much! $\endgroup$
    – Dave
    Commented Mar 16, 2023 at 1:32
  • 2
    $\begingroup$ I think your question remains legitimate: why can't this model fit in-sample at least as well as a model that has no features? $\endgroup$
    – Dave
    Commented Mar 16, 2023 at 1:48

1 Answer 1

1
$\begingroup$

(In the comments, I see you think it solved, so this is more general advice.)

To identify the root cause of a poor model, you should start by getting a baseline model to compare against.

As you are using H2O, that is very easy, as you can just swap h2o.randomForest with, say, h2o.glm (for generalized linear model). You could also try h2o.deeplearning for a neural net, but using a linear model as a baseline is often good as it is quick, and the defaults are good.

So if the results are equally bad, the data or how it is loaded becomes the suspect. If the glm is much better, the way you are using randomForest becomes suspect.

With H2O, you have H2O Flow: open a web browser at http://localhost:54321/ and you can see the models and data you have loaded in, and you can then explore and analyze it.

Another thing, which also came up in the comments, is that if you are overfitting you should be seeing good scores when evaluating on your training data. If you don't then I'd be suspicious of the data again.

E.g. with this data, the very best training model score you could hope for is 0.33.

x1,x2,y
1,2,A
2,3,A
1,2,B
2,3,B
1,2,C
2,3,C

This kind of thing can happen if the data has been loaded badly, e.g. it thought the data was csv, but it was actually semi-colon separated. Again, viewing the data in H2O Flow can confirm if this has happened.

$\endgroup$
1
  • $\begingroup$ Thank you very much Darren! This is very helpful I had no idea H2O Flow even existed. $\endgroup$ Commented Mar 17, 2023 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.