I have produced a series of random forest regression models (13 in total) and would like to check if the models are significantly different. I have already compared the models using summary statistics (eg. MSE, MAE, RMSE, nRMSE).

The models were developed using the same features and observations. However, the values corresponding to the features vary between the models. Without going into unnecessary detail, the features were produced by an algorithm which can be adjusted to control the sensitivity. I am comparing models produced at different sensitivities to see if it has a meaningful effect on the regression results.

The differences in RMSE between the models is not very large and I'd like to check if is difference is significant. However, I'm not sure what the best practices are for comparing a series of non-parametric models such as Random Forests. Would an equivalence test be appropriate? Or, is there another, more appropriate test that should be used instead?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.