I'm training a binary classifier using time series cross-validation and a rolling window approach, resulting in $k$ test sets. I want to construct a cumulative uplift curve to evaluate the model.
I've considered simply pooling all the test sets together, but that could result in the same customer appearing $k$ times, which isn't ideal. On the other hand, creating $k$ separate uplift curves would be cumbersome.
I would appreciate any advice or insights on the best practices to handle this issue.
To create a cumulative uplift curve for time series CV with rolling windows, I used the following method:
Divide each day's test set into equal percentiles based on classifier predictions.
Calculate daily uplift for each percentile.
Average the uplifts across days for each percentile.
This gives an averaged cumulative uplift curve without pooling test sets or repeating customers.
However, there are assumptions in my approach. For example, I assume that each day (or window) has roughly the same distribution. I haven't found literature supporting this method, and I welcome feedback on its validity.
