Is my model overfitting?

Question

I am currently working on a very imbalanced dataset:

• 24 million transactions (rows of data)
• 30,000 fraudulent transactions (0.1% of total transactions)
  

The dataset is split via Year, into three sets of training, validation and test. I am using XGBoost as the model to predict whether a transaction is fraudulent or not. After tuning some hyperparameters via optuna, I have received such results

Model parameters and loss

from sklearn.metrics import accuracy_score, classification_report, precision_score, recall_score, f1_score, roc_auc_score, precision_recall_curve, auc, average_precision_score, ConfusionMatrixDisplay, confusion_matrix
import matplotlib.pyplot as plt
evalset = [(train_X, train_y), (val_X,val_y)]

params = {'lambda': 4.056095667860487, 'alpha': 2.860539790760471, 'colsample_bytree': 0.4, 'subsample': 1, 'learning_rate': 0.03, 'n_estimators': 300, 'max_depth': 44, 'random_state': 42, 'min_child_weight': 27}
model = xgb.XGBClassifier(**params, scale_pos_weight = estimate, tree_method = "gpu_hist")  
model.fit(train_X,train_y,verbose = 10, eval_metric='logloss', eval_set=evalset)

[0] validation_0-logloss:0.66446    validation_1-logloss:0.66450
[10]    validation_0-logloss:0.45427    validation_1-logloss:0.45036
[20]    validation_0-logloss:0.32225    validation_1-logloss:0.31836
[30]    validation_0-logloss:0.23406    validation_1-logloss:0.22862
[40]    validation_0-logloss:0.17265    validation_1-logloss:0.16726
[50]    validation_0-logloss:0.13003    validation_1-logloss:0.12363
[60]    validation_0-logloss:0.09801    validation_1-logloss:0.09230
[70]    validation_0-logloss:0.07546    validation_1-logloss:0.06987
[80]    validation_0-logloss:0.05857    validation_1-logloss:0.05278
[90]    validation_0-logloss:0.04581    validation_1-logloss:0.04001
[100]   validation_0-logloss:0.03605    validation_1-logloss:0.03058
[110]   validation_0-logloss:0.02911    validation_1-logloss:0.02373
[120]   validation_0-logloss:0.02364    validation_1-logloss:0.01859
[130]   validation_0-logloss:0.01966    validation_1-logloss:0.01472
[140]   validation_0-logloss:0.01624    validation_1-logloss:0.01172
[150]   validation_0-logloss:0.01340    validation_1-logloss:0.00927
[160]   validation_0-logloss:0.01120    validation_1-logloss:0.00752
[170]   validation_0-logloss:0.00959    validation_1-logloss:0.00616
[180]   validation_0-logloss:0.00839    validation_1-logloss:0.00515
[190]   validation_0-logloss:0.00725    validation_1-logloss:0.00429
[200]   validation_0-logloss:0.00647    validation_1-logloss:0.00370
[210]   validation_0-logloss:0.00580    validation_1-logloss:0.00324
[220]   validation_0-logloss:0.00520    validation_1-logloss:0.00284
[230]   validation_0-logloss:0.00468    validation_1-logloss:0.00253
[240]   validation_0-logloss:0.00429    validation_1-logloss:0.00226
[250]   validation_0-logloss:0.00391    validation_1-logloss:0.00205
[260]   validation_0-logloss:0.00362    validation_1-logloss:0.00191
[270]   validation_0-logloss:0.00336    validation_1-logloss:0.00180
[280]   validation_0-logloss:0.00313    validation_1-logloss:0.00171
[290]   validation_0-logloss:0.00291    validation_1-logloss:0.00165
[299]   validation_0-logloss:0.00276    validation_1-logloss:0.00161

Learning curve

F1 and PR AUC scores

F1 Score on Training Data : 0.8489783532267853
F1 Score on Test Data : 0.7865990990990992
PR AUC score on Training Data : 0.9996174980952233
PR AUC score on Test Data : 0.9174896435002448

Classification reports of training/testing sets

Training report
              precision    recall  f1-score   support

           0       1.00      1.00      1.00  20579668
           1       0.74      1.00      0.85     25179

    accuracy                           1.00  20604847
   macro avg       0.87      1.00      0.92  20604847
weighted avg       1.00      1.00      1.00  20604847

Test report
              precision    recall  f1-score   support

           0       1.00      1.00      1.00   2058351
           1       0.95      0.67      0.79      2087

    accuracy                           1.00   2060438
   macro avg       0.98      0.83      0.89   2060438
weighted avg       1.00      1.00      1.00   2060438

Confusion matrices (1st is training set, 2nd is testing set)

I see that my PRAUC of the training dataset is nearly 1 and it has perfect recall score, so I suspect that my model is overfitting. However, when I test these results on a validation set and testing set, the results are not too far off, and still achieve what I believe to be decent scores.

I would love to hear your thoughts on this, and thank you all in advance and I would appreciate any response!

Answer 1

From an ML perspective, there's not really a precise definition of whether you've overfit or not. It's also not binary, all models have picked up some signal at the expense of some noise, and are thus somewhat overfit. Being a little overfit is the cost of learning.

The delta between training and test performance is a good guide, but a nice (admittedly pathological) example of where this will give you incorrect results are Random Forests which intentionally will get very good training loss knowing that this loss won't be achievable on a test set. Nonetheless, a random forest might allow you to achieve better test loss than another model which has a smaller gap between train and test loss, and in this case the Random Forest is the better model.

Looking at the output from your training process, your validation loss has gone down for the entirety of the training process... so you've probably got the best you could have got out of that particular model setup. I guess I would categorically say you'd overfit if you had overshot the optimal point and your validation loss had started to go up. But you could have another model which you'd fit optimally (i.e. stopped training when your validation loss stopped going down), which had a bigger/smaller gap between training and validation loss, but the all-important metric is which of the two models performs better on the test set.

So I personally think the question of "have I overfit" isn't quite the right one. You should always use the model that performs best on the test set, whilst taking great care to make sure your test is valid and you've not leaked information from the training into the test set or used information you wouldn't have available to you in prod (e.g. data that theoretically exists but only comes back from the data provider in a batch ever 24hrs and thus not available at the time you want to make the prediction)

Also, seeing as you mention some of the metrics you've used, some thoughts on this: Fundamentally, the problem you're trying to solve is "should I deploy this model" so you want to understand whether it's sufficiently better than your current baseline (be that an ML model or just a business process, which could be "do nothing") so as to warrant putting into production. To that end, metrics like F1-score, or AUC aren't really what you need.

When it comes to fraud, it would be typical to have an estimate of what a false positive costs you (presumably every predicted positive leads to an extra level of verification being introduced which leads to some dropoff, and you can make some assumptions on the typical range / maybe you have some data on this), as well as what a false negative costs you (essentially the cost of having to reimburse the transaction I would imagine)

Then, at different thresholds, you can quantify how much money you're losing to the combination of fraud + dropouts and compare that to your current procedure. That will lead you to a number of how much money deploying this model could make/lose you. [Disclaimer: it's a little more complicated than this, there are secondary metrics, reputational concerns etc, this was more meant to be indicative]