I've read through the following posts that answered the question I was going to ask:

Use Random Forest model to make predictions from sensor data

Decision tree for output prediction

Here's what I've done so far: I compared Logistic Regression to Random Forests and RF outperformed Logistic. Now the medical researchers I work with want to turn my RF results into a medical diagnostic tool. For example:

If you are an Asian Male between 25 and 35, have Vitamin D below xx and Blood Pressure above xx, you have a 76% chance of developing disease xxx.

However, RF doesn't lend itself to simple mathematical equations (see above links). So here's my question: what ideas do you all have for using RF to develop a diagnostic tool (without having to export hundreds of trees).

Here's a few of my ideas:

  1. Use RF for variable selection, then use Logistic (using all possible interactions) to make the diagnostic equation.
  2. Somehow aggregate the RF forest into one "mega-tree," that somehow averages the node splits across trees.
  3. Similar to #2 and #1, use RF to select variables (say m variables total), then build hundreds of classification trees, all of which uses every m variable, then pick the best single tree.

Any other ideas? Also, doing #1 is easy, but any ideas on how to implement #2 and #3?

  • Make a prediction equation for each tree (it will be simple split points), and then average the predictions from each equation? You'll get one monster equation, but it will fully represent the forest. – Zach Oct 8 '13 at 18:56
  • Good idea @Zach. But unfortunately I'm trying to avoid anything "monster." – dfife Oct 8 '13 at 18:57
  • 1
    Would you mind restating why random forests pose implementation issues for you? They're not particularly intensive unless you have thousands of features. You could prune it down, but it's unlikely you'll have an analytic form that's digestible. – Jessica Mick Oct 8 '13 at 19:14
  • @Jacob--The problem is that RF has lots of decision trees. I'd love to report a single formula (< a few lines long if possible) that predicts nearly as accurately as RF. Since I'm publishing my work to an audience of modest statistical sophistication, I think exporting pages upon pages of trees would severely limit the probability of my findings becoming implemented in clinical settings. – dfife Oct 8 '13 at 19:38
up vote 6 down vote accepted

Here there are some thoughts:

  1. All black-box models might be inspected in some way. You can compute the variable importance for each feature for example or you can also plot the predicted response and the actual one for each feature (link);
  2. You might think about some pruning of the ensemble. Not all the trees in the forest are necessary and you might use just a few. Paper: [Search for the Smallest Random Forest, Zhang]. Otherwise just Google "ensemble pruning", and have a look at "Ensemble Methods: Foundations and Algorithms " Chapter 6;
  3. You can build a single model by feature selection as you said. Otherwise you can also try to use Domingos' method in [Knowledge acquisition from examples via multiple models] that consists in building a new dataset with black-box predictions and build a decision tree on top of it.
  4. As mentioned in this Stack Exchange's answer, a tree model might seem interpretable but it is prone to high changes just because of small perturbations of the training data. Thus, it is better to use a black-box model. The final aim of an end user is to understand why a new record is classified as a particular class. You might think about some feature importances just for that particular record.

I would go for 1. or 2.

  • Very thorough answer. Thanks! – dfife Oct 10 '13 at 13:29

I've had to deal with the same situation of using RF in a diagnostic setting, with stakeholders who are used to algorithms that boil down to a single, readable equation. I've found that if you start by explaining a simple decision tree (here you can use equations), then a very complicated one, and then explain the drawbacks of over-fitting, you start to get some head nods. Once you explain that many small trees can mitigate inaccuracy by being grown differently ("random"), and that they can be taken as an ensemble vote or average to avoid over-fitting but still account for edge cases, you get understanding. Here are some example slides I've used with good reception:

You can't get away from trees in a forest, and they are what give the algorithm so much predictive power and robustness, so there is rarely a better solution if RF is working very well for you. Ones that will compare, like SVM (depending on your data), will be just as complex. You have to make them understand that any good solution is going to be a black box of sorts (to the user). Your best move is to create a consumable implementation that doesn't require any more effort than a single equation would. I've had success with building an RF model in Python (via sklearn), and creating a simple web server REST API that loads that model into memory and accepts the variables in a POST to output the prediction. You can also do this in Java or R very easily, or skip the API and just create an executable binary/jar that takes the data as arguments.

  • Good point!--Aggregating the results of RF into a single equation will inevitably lose some of its advantages. I hadn't thought of that. – dfife Oct 9 '13 at 12:51
  • 2
    Most likely, yes. However, using RF to compute variable importance to reduce dimensionality, followed by a method your audience is more familiar with, is a common compromise. It's important to know that variable importance in RF is generally computed based on the performance within the RF (lift, reduce error, etc.) and might not all carry similar weight in other methods. – wwwslinger Oct 9 '13 at 15:11

I have experience of deploying random forests in a SQL Server environment via User Defined Function. The trick is to convert the IF-THEN ELSE rules that you get from each tree into a CASE-WHEN END or any other Conditional Processing construct (admittedly I've used JMP Pro's Bootstrap Forest implementation - 500k lines of SQL code).

There is absolutely no reason why this cannot be achived using the rattle R package. Have a look at randomForest2Rules & printRandomForests functions in that package. Both take random forest object as input and visit each tree in the forest and output a set of IF-THEN ELSE rules. Taking this as a starting point it should not be difficult converting this logic into your desired language in an automated way, since the output from the above mentioned function is structured text.

The above, also makes it important to decide the smallest no. of trees you need in the forest to make predictions at a desired level of accuracy (hint: plot(rf.object) shows you at what point the forest predictions do not improve despite adding more trees.) in order to keep the no. of lines to represent the forest down.

I have written a function to generate SQL code for a random forest model, see: https://gist.github.com/shanebutler

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • 3
    At present, this is more of a comment than an answer. Would you mind expanding it into more of an answer? – gung Aug 10 '14 at 3:21

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.