Analysis for checking if an Ensemble model is a better fit for a dataset than Primitive model I have a dataset and have the option to apply either GLM (primitive) or a Random Forest (ensemble). So far the Random Forest is giving way better results than the GLM. As it is generally believed that ensemble models should not be used unless absolutely necessary, hence I am looking for any analysis which I could perform on the dataset, which could prove that indeed the only way/better way to model the relationship between variables in the dataset is by using a ensemble model like Random Forest etc.
 A: If you want to argue that your Random Forest is generating better predictions than your linear model, then you could just show that (for example) your out-of-sample RMSE is lower for your random forest than for your linear model--it's as simple as that.
The primary downside of using a non-linear model instead of a plain-vanilla linear model is that, by and large, as your methods become more sophisticated, your resulting models will become more opaque (i.e. harder to interpret). If your goal is pure prediction then this won't matter. However, if you're trying to do some statistical inference then it's a different story.
A: *

*As it is generally believed that non-linear models should not be used unless absolutely necessary. I don't think this is accurate. 

*GLMs are fairly flexible. Are you including regression splines, interactions... in your model? 

*Regularization is often used with GLMs - the number of parameters don't need to be fixed. 

*As mentioned by Scortchi, I don't think you can prove that any linear model will be outperformed by random forest - given the large number of options available. 

