I am training a binary classifier using random forest. I'm interested in a clinical event, and my raw data is longitudinal electronic medical records. The goal for this classifier is to be able to detect patients who are at risk for the event based on their recent clinical records. To do this, I am looking back 90 days from the date of the encounter in which the event was recorded for my "cases" (patients who experienced the event), and looking back 90 days from a randomly sampled encounter in which the event was NOT recorded for my "controls" (patients who never experienced the event).

This results in a dataset that is enriched with positive cases since it collapses events over several years of follow-up, whereas in "live" shorter-term data, the incidence is much lower.

What I would like to do is use this classifier to obtain predicted probabilities for patients in real-time based on their most recent 90-days of clinical records. I know that if you train on data that has a certain class balance, and test on data that has another, you can expect very different performance metrics, since they are based on dataset-level factors (class balance). But is it safe to interpret the predicted probability for individual cases in this context?



1 Answer 1


Firstly, you would need to implement a Random Survival Forest approach, not Random Forest -- the R-project has some very good modules for this. If you stick with time-to-event modeling (for which each subject has a yes/no failure variable as well as a time variable) and RSF, you can extract predicted probabilities since the entire RSF algorithms are survival-based. However, I wouldn't implement RSF before looking at results from Kaplan-Meier analysis and Cox PH regression first.

Looking deeper at your question, the strength of RF and RSF are really to determine which variables (features) are important predictors of the outcome. Cutpoint values of the important features provided by e.g. CART are not in theory a mainstay of RF and RSF because for a default run, RF/RSF will generate e.g. 5000 bootstraps of the data (called in-bag samples) having sample size $n$, train each in-bag sample and generate a decision tree using randomly drawn variables to split cases within each parent node (node splitting), until nodes cannot be split anymore. For each out-of-bag (OOB) sample of cases not selected during generation of each bootstrap $(n_{oob} \sim 0.37n)$, OOB cases are dropped down each trained tree to determine prediction accuracy.

It's common to average the prediction accuracy over the 5000 trees, but one must recall that the features are randomly selected to make each node split within each tree. And therefore, since the objects are different across each trained tree (each bootstrap has different objects), the optimal cutpoints used for each feature at a given node split within a tree is specific to the combination of the randomly used feature and the cases in the given bootstrap. When all is done, there is no uniform set of cutpoint values for the variables used -- since they change over each tree.

In summary, RF and RSF are I believe some of the better methods available, but fundamentally I employ them to inform me which features are most informative. I wouldn't operationalize an RF or RSF to sit on a server and make predictions for patients -- this can be better done via ensemble classifier fusion (see Ludmila Kuncheva's work).

  • $\begingroup$ Thanks for your thorough response. Can you tell me a little more about what you mean when you say you wouldn't implement RSF before looking at results from Kaplan-Meier and CoxPH? What should one look for before proceeding? $\endgroup$
    – ebnone
    Commented Apr 15, 2017 at 2:56
  • $\begingroup$ KM and Cox PH regression are long-standing methods that have been employed for what you want to do. You should always determine if and whether vanilla-flavored methods fail, before using RSF. Cox PH is a regression model that can generate a probability for each new patient. $\endgroup$
    – user32398
    Commented Apr 15, 2017 at 3:08
  • $\begingroup$ Got it, thanks. I started with Cox PH - didn't work well due to proportional hazards violations and some other issues with the way the certain time-varying indicators were generated. $\endgroup$
    – ebnone
    Commented Apr 15, 2017 at 3:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.