Here's the scenario, slightly altered to a common one.

Credit card fraud, payments for the last 12 months (a rolling window). Train with the data from the first 10 months, validate with data from the 11th and test with the data from the 12th month.

My rationale for this is that when used for real, we'll always use the history (be it of the same card or everything in the past, like fraud patterns).

Are there any methodological problems with this approach?

  • $\begingroup$ Then you'll slide the window one month and do it again with the last as test? $\endgroup$ – Meadowlark Bradsher Jun 4 '14 at 1:08
  • $\begingroup$ Yes, from scratch (the idea being that only the last year matters). $\endgroup$ – wishihadabettername Jun 4 '14 at 2:53
  • $\begingroup$ What is the prediction target? $\endgroup$ – Daniel Jun 4 '14 at 6:50
  • $\begingroup$ a binary field. Fraud or not. With whatever models are found to be working: logistic regression, SVM and so on. $\endgroup$ – wishihadabettername Jun 4 '14 at 8:59
  • $\begingroup$ How many records do you have in the first 10 months? More importantly, how many fraud records are there in the first 10 months? I will be able to assist with this information! Thanks! $\endgroup$ – Matt Reichenbach Jun 4 '14 at 11:43

Your approach is statistically valid.

With that being said, you may also want to split the first 10 months into a training and validation set. Build a logistic regression model using only the training set or perform cross-validation to obtain the lowest misclassification error rate in the validation set. This way you are less likely to overfit the first 10 months of data, which will likely lead to better predictions in the 11th month (your validation set), the 12 month (your test set), and most importantly, future months. As with any predictive model, it will be important to see how the model holds up over time!

Please let me know if you need any further help! Happy modeling!

  • $\begingroup$ Indeed, I will over sample (and look at other approaches for imbalanced classes) $\endgroup$ – wishihadabettername Jun 4 '14 at 12:09
  • 1
    $\begingroup$ I think the interest is more in risk than in classification. $\endgroup$ – Frank Harrell Jun 4 '14 at 12:15

What specifically do you want the validation (11th month) and test (12th months) sets for?

  • If you do any kind of optimization with the 11th month data, you are right that you need to test you final model on independent cases. However, the 12th month would not be independent for that scenario: it has an overlap of 10 months with the training and optimization (11th mondth) data. In order to have independent data for testing, you'd need to split the records into training and optimization on the one hand and records kept apart for validation/testing of the final optimized model (test on the 11th month of independent cards).

  • The 12th month of the same credit cards where you used the first 10 months for training would be suited e.g. for testing how far into the future you can predict the fraud risk (i.e. how fast the quality of your predictions deteriorates)

  • $\begingroup$ Thanks @cbeleites. The card number is changed once fraud is noticed by the customer and it's not used as a predictor. What the training is after is the patterns of fraud. The 12th month is with new cases of fraud that, often, have the same patterns. There is no other relationship with the past than the patterns. Thus months 1-10 are to extract the patterns, 11 is for validation and tuning and 12 for the test. $\endgroup$ – wishihadabettername Jun 5 '14 at 6:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.