Statistical models in general are not well suited for applications outside the training range.

Using a random forest binary classifier aimed at detecting an event, I've trained it using 100 data points due to data limitation. I then applied it in a similar dataset, which had 2000 data points (but no y values to train with) to predict the number of events. Despite having similar statistical properties such as mean and standard deviation, the larger datasets implies in more extreme values (tails). In fact, 5% of the larger dataset lies outside the training range.

That said, for this specific case of binary prediction, I reckon it could be ill-posed to rely on the RF model. However, does anybody have any suggestions or alternatives (parametric/linear models, tools, packages) on how to deal with cases like this?

  • $\begingroup$ Take a look at semi-supervised approaches: "few labeled - lots unlabeled" $\endgroup$ Commented May 7, 2021 at 13:16
  • 1
    $\begingroup$ Thank you for the suggestion, I will take a look at it right away. I ran this sort of test in the data basically converting every exceeding value to the respective maximum or minimum value of the training set and the results did not change (it is a classifier based on thresholds after all). Would this be acceptable somehow? $\endgroup$
    – Henrique
    Commented May 7, 2021 at 18:19