3
$\begingroup$

I have a dataset of the following form:

client_id | date       | client_attr_1 | client_attr_2 | client_attr3 | money_spend
1         | 2020-01-01 |           123 |           321 |          188 |      150.24
1         | 2020-01-02 |           123 |           321 |          188 |       18.25
1         | 2020-01-03 |           123 |           321 |          188 |       12.34
2         | 2020-01-02 |           233 |           421 |          181 |       10.10
2         | 2020-01-03 |           233 |           421 |          181 |       20.00
2         | 2020-01-04 |           233 |           421 |          181 |       11.12
2         | 2020-01-01 |           233 |           421 |          181 |       18.36
3         | 2020-02-01 |           723 |           301 |          255 |        1.14
3         | 2020-02-01 |           723 |           301 |          255 |        1.19

My goal is to predict money spend for new clients, day by day.

The goal of the validation procedure is to get a model performance that is not biased by group/time leakage.

I can imagine that an ideal validation scheme that would reflect the actual prediction time situation for that problem would take the following into account:

  1. Groups - clients, ensure client's observations are not in train and validation sets at the same time.
  2. Time - make sure that the model is not training on future clients and predicting on clients from the past to avoid look-ahead bias.

I find it a bit inconvenient as it requires implementing custom validation procedure that could cause some additional problems (e.g. highly different train/test sizes with repeated validation). Therefore, I'd like to drop the second assumption. For that to be a reasonable thing to do, I believe that what I need to check is whether the actual time series (spend given date) are somehow dependent (correlated) on the same dates for different clients (I assume it will not be the case).

Now the questions are:

  1. Is it the right thing to check?
  2. Is comparing time series of different clients on the same dates enough?
  3. Is there a better/proper way to asses such dependency?
  4. Perhaps I need not to validate that or anything else for the reasons I'm not seeing?
$\endgroup$

1 Answer 1

1
$\begingroup$

You can look at this answer for a similar question.

The solution relies on, in a first step, splitting clients on training and testing, say you have clients [0, 1, 2, 3]

  • Split 1 : Train : [0, 1] Test : [2, 3]
  • Split 2 : Train : [0, 2] Test : [1, 3]
  • ...

Then, for each training/testing split, split again in a times series way :

  • Split 1a : Train on the 6 first month for customer [0, 1] and test on the 7th month for customers [2, 3]
  • Split 1b : Train on the 7 first month for customer [0, 1] and test on the 8th month for customers [2, 3]
  • Split 1c : Train on the 8 first month for customer [0, 1] and test on the 9th month for customers [2, 3]
  • ...
  • Split 2a : Train on the 6 first month for customer [0, 2] and test on the 7th month for customers [1, 3]
  • Split 2b : Train on the 7 first month for customer [0, 2] and test on the 8th month for customers [1, 3]
  • Split 2c : Train on the 8 first month for customer [0, 2] and test on the 9th month for customers [1, 3]
  • ...

You also have the implementation of the solution production a cv object that you can use in sklearn (ex: gridsearchCV).

$\endgroup$
3
  • $\begingroup$ Thank you, indeed the problem there is exactly the same as my problem. I have a quite different question though - I'd like to omit the time series based part, and I'm asking how to properly assess whether this can be done without harming validation error too much. $\endgroup$
    – Matek
    Aug 31, 2020 at 20:25
  • $\begingroup$ You can use a K-fold for non-overlapping groups, suggested in sklearn here and here for implementation details. $\endgroup$
    – SoufianeK
    Aug 31, 2020 at 20:33
  • $\begingroup$ Yes I can, but firstly I need to properly assess whether that will not introduce a look-ahead bias and that's what this question is about :) - how to check it properly $\endgroup$
    – Matek
    Aug 31, 2020 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.