2
$\begingroup$

I have been working on binary classification problem using algorithms such as Random Forest, Boosting methods, neural networks and logistic regression. I have data from Jan 2017 to Jan 2022. We wish to train the model based on historical (completed transactions) from Jan 2017 to Jan 2020. And use this model to predict the outcome of active transactions (from Feb 2020 to Jan 2022).

I tried the below two approaches for train test split

a) usual sklearn train_test_split (random)

b) manual train test split (time-based) - all records from 2017 t0 2020 Jan were train and all records from Feb 2020 to Jan 2022 were Test. I use dataframe filter to filter records based on year value.

However, I found out that my performance degraded when I chose time based split (option b above) even for train data.

whereas regular sklearn train_test_split gave better performance.

Can I know why does this happen? Has anyone here encountered this sort of behavior?

How can we avoid this? and what would be the right way to split the data?

$\endgroup$

1 Answer 1

3
$\begingroup$

Speaking generally, and noting as an aside that data splitting is a bad idea unless you have > 20,000 observations, splitting on time represents a missed opportunity for modeling time trends. To say that a model doesn't validate in a later time period may just mean that there was a time trend that was ignored in model develop. Time can be a very important variable, and one that needs to be modeled as a continuous but nonlinear effect. Rather than splitting on time or place I like to model the effects of time and place.

$\endgroup$
5
  • $\begingroup$ Thanks for the response. I have only 977 observations. Do you mean data splitting is bad for < 20000 records? If yes, how do I create train and test sets? Or do you mean data splitting based on time is bad for < 20K records? but yes, your response helped me understand the reason for performance drop better. $\endgroup$
    – The Great
    Feb 25, 2022 at 13:05
  • $\begingroup$ Data splitting needs about 20,000 independent observations, otherwise you'll find that if you split again you'll get different models and different performance estimates. Don't think about distinct training and test sets. Use resampling to estimate the likely future performance of the model, as described at length on this site. E.g. use 100 repeats of 10-fold cross-validation or bootstrap 400 resamples. $\endgroup$ Feb 25, 2022 at 13:33
  • $\begingroup$ Of the approaches the OP mentions, only regression can model the effect of time explicitly; RFs, GBMs and NNs don't, at least not explicitly. So does this imply that in the small-data regime with a time component, it might be better to give up on "flexibility" and commit to a (nonlinear but precise) formula for the time effect? $\endgroup$
    – dipetkov
    Mar 27, 2022 at 16:12
  • 1
    $\begingroup$ @FrankHarrell You wrote "..., as described at length on this site." Can you provide the link? $\endgroup$
    – T_T
    Feb 3, 2023 at 1:17
  • 1
    $\begingroup$ hbiostat.org/bbr/… $\endgroup$ Feb 3, 2023 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.