How to compare two random forests in scikit-learn?

Question

With most learning algorithms, one can compare the models resulting from applying the algorithm on samples of data by the parameters of the models. For example, one can compare two logistic regression models by comparing the learned model parameters (I'm not referring to the hyperparameters here).

I would like to do that with two random forest models trained with scikit-learn's random forest algorithm. However, I do not see any properties or methods that give me access to parameters that can be used to draw a comparison.

How would you compare two random forest models?

Answer 1

In logistic regression, the coefficients of a variable indicates the expected change in log odds of observed outcome, per unit change in that variable. So by looking at these model parameters, one can get a feeling which variables are most crucial in increasing/decreasing the probability of observing the outcome.

Something similar in random forest is the feature importance. In scikit-learn, it is possible to extract the mean decrease in impurity for each feature. So when this value is large, it means that splitting on this feature will on average more likely result in pure groups. It resembles large positive or large negative coefficients in the logistic setting.

https://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html

Answer 2

With most learning algorithms, one can compare the models resulting from applying the algorithm on samples of data by the parameters of the models.

This claim is not true in general. There are models with thousands, or millions, of parameters like neural networks where there is no meaningful way of comparing the models based on the parameters. There are nonparametric models with a theoretically infinite number of parameters. In fact, you could meaningfully compare only the simplest models, like small linear models, in terms of parameters.

Random forest is a collection of decision trees that do not have parameters per se. You could plot all the trees from one random forest and compare them to another, but this would mean comparing hundreds of trees to each other.

Answer 3

1. For the overall fit, any binary models can be compared, irrespective of how they were trained, using some statistics which are computed based on the model outcomes. For all these measures it is better to use probabilities rather than binary class predictions, i.e. Prob( class = 1| predictors).

a) ROC-AUC. In R you can get ROC-AUC with confidence interval, so can do a statistical test if one model is >> other in how it separates two classes, in scikit I didnt check, but you can try bootstrap to get confidence intervals.

b) Second measure is R2 for binary outcomes - scaled Brier score, it is less common, but more powerful measure of the overall model fit. You can google it, but essentially it is just R2 for a regression, where predicted y is model's probabilities and outcome is 1/0 - true class.

2. If you are interested in the impact factors of each of the risk factor, you can use variable importance impact as suggested above, or also use SHAP library. It plots the size and sign of each of the risk factors for each observation in the training set in one graph, you can compare these graphs for your random forests. https://towardsdatascience.com/explain-any-models-with-the-shap-values-use-the-kernelexplainer-79de9464897a

3. You can compute predicted odds ratio yourself, e.g. take a median observation with all risk factors set to means, compute predicted probability of class 1, then change a risk factor by 1sd and see the change in prediction. You can compute odds ratios for each risk factor and compare those between your random forests (or any other models). Ideally, you want to bootstrap to get confidence intervals for those ORs. Note, that in logistic regression these ORs will be the same across all observations, not just median ones, in RF those can vary a lot depending on what values you set for other parameters - so here you may need to see ORs for what group of people (median or some others) you want to compute and compare between the models. The advantage of computing OR by hand is that you can compute for the target group, and also change ORs to absolute risk change or any other outcome measure that you want to compare.