I am training a binary classifier using random forest. I'm interested in a clinical event, and my raw data is longitudinal electronic medical records. The goal for this classifier is to be able to detect patients who are at risk for the event based on their recent clinical records. To do this, I am looking back 90 days from the date of the encounter in which the event was recorded for my "cases" (patients who experienced the event), and looking back 90 days from a randomly sampled encounter in which the event was NOT recorded for my "controls" (patients who never experienced the event).
This results in a dataset that is enriched with positive cases since it collapses events over several years of follow-up, whereas in "live" shorter-term data, the incidence is much lower.
What I would like to do is use this classifier to obtain predicted probabilities for patients in real-time based on their most recent 90-days of clinical records. I know that if you train on data that has a certain class balance, and test on data that has another, you can expect very different performance metrics, since they are based on dataset-level factors (class balance). But is it safe to interpret the predicted probability for individual cases in this context?