Statistical models in general are not well suited for applications outside the training range.
Using a random forest binary classifier aimed at detecting an event, I've trained it using 100 data points due to data limitation. I then applied it in a similar dataset, which had 2000 data points (but no y values to train with) to predict the number of events. Despite having similar statistical properties such as mean and standard deviation, the larger datasets implies in more extreme values (tails). In fact, 5% of the larger dataset lies outside the training range.
That said, for this specific case of binary prediction, I reckon it could be ill-posed to rely on the RF model. However, does anybody have any suggestions or alternatives (parametric/linear models, tools, packages) on how to deal with cases like this?