Ideas for outputting a prediction equation for Random Forests

Question

I've read through the following posts that answered the question I was going to ask:

Use Random Forest model to make predictions from sensor data

Decision tree for output prediction

Here's what I've done so far: I compared Logistic Regression to Random Forests and RF outperformed Logistic. Now the medical researchers I work with want to turn my RF results into a medical diagnostic tool. For example:

If you are an Asian Male between 25 and 35, have Vitamin D below xx and Blood Pressure above xx, you have a 76% chance of developing disease xxx.

However, RF doesn't lend itself to simple mathematical equations (see above links). So here's my question: what ideas do you all have for using RF to develop a diagnostic tool (without having to export hundreds of trees).

Here's a few of my ideas:

1. Use RF for variable selection, then use Logistic (using all possible interactions) to make the diagnostic equation.
2. Somehow aggregate the RF forest into one "mega-tree," that somehow averages the node splits across trees.
3. Similar to #2 and #1, use RF to select variables (say m variables total), then build hundreds of classification trees, all of which uses every m variable, then pick the best single tree.

Any other ideas? Also, doing #1 is easy, but any ideas on how to implement #2 and #3?

Answer 1

Here there are some thoughts:

1. All black-box models might be inspected in some way. You can compute the variable importance for each feature for example or you can also plot the predicted response and the actual one for each feature (link);
2. You might think about some pruning of the ensemble. Not all the trees in the forest are necessary and you might use just a few. Paper: [Search for the Smallest Random Forest, Zhang]. Otherwise just Google "ensemble pruning", and have a look at "Ensemble Methods: Foundations and Algorithms " Chapter 6;
3. You can build a single model by feature selection as you said. Otherwise you can also try to use Domingos' method in [Knowledge acquisition from examples via multiple models] that consists in building a new dataset with black-box predictions and build a decision tree on top of it.
4. As mentioned in this Stack Exchange's answer, a tree model might seem interpretable but it is prone to high changes just because of small perturbations of the training data. Thus, it is better to use a black-box model. The final aim of an end user is to understand why a new record is classified as a particular class. You might think about some feature importances just for that particular record.

I would go for 1. or 2.

Answer 2

I've had to deal with the same situation of using RF in a diagnostic setting, with stakeholders who are used to algorithms that boil down to a single, readable equation. I've found that if you start by explaining a simple decision tree (here you can use equations), then a very complicated one, and then explain the drawbacks of over-fitting, you start to get some head nods. Once you explain that many small trees can mitigate inaccuracy by being grown differently ("random"), and that they can be taken as an ensemble vote or average to avoid over-fitting but still account for edge cases, you get understanding. Here are some example slides I've used with good reception:

You can't get away from trees in a forest, and they are what give the algorithm so much predictive power and robustness, so there is rarely a better solution if RF is working very well for you. Ones that will compare, like SVM (depending on your data), will be just as complex. You have to make them understand that any good solution is going to be a black box of sorts (to the user). Your best move is to create a consumable implementation that doesn't require any more effort than a single equation would. I've had success with building an RF model in Python (via sklearn), and creating a simple web server REST API that loads that model into memory and accepts the variables in a POST to output the prediction. You can also do this in Java or R very easily, or skip the API and just create an executable binary/jar that takes the data as arguments.

Answer 3

I have experience of deploying random forests in a SQL Server environment via User Defined Function. The trick is to convert the IF-THEN ELSE rules that you get from each tree into a CASE-WHEN END or any other Conditional Processing construct (admittedly I've used JMP Pro's Bootstrap Forest implementation - 500k lines of SQL code).

There is absolutely no reason why this cannot be achived using the rattle R package. Have a look at randomForest2Rules & printRandomForests functions in that package. Both take random forest object as input and visit each tree in the forest and output a set of IF-THEN ELSE rules. Taking this as a starting point it should not be difficult converting this logic into your desired language in an automated way, since the output from the above mentioned function is structured text.

The above, also makes it important to decide the smallest no. of trees you need in the forest to make predictions at a desired level of accuracy (hint: plot(rf.object) shows you at what point the forest predictions do not improve despite adding more trees.) in order to keep the no. of lines to represent the forest down.