0
$\begingroup$

I have a project for predicting credit card approvals (binary classification).

I got stuck on feature selection, hyperparameter tuning and final testing stages, as I don't know how to properly modify (balance) data to avoid data leakage or overfitting, as my datasets are imbalanced


Sets look as follows:

  1. training_set (highly imbalanced, 112 000 samples ~)

  2. testing_set (highly imbalanced, 55 000 samples ~)

  3. validation set (highly imbalanced, 55 000 samples ~ )

All sets has been preprocessed independently.

As a model I decided to choose XGBoost and f1 score as evaluation metric.


What I want to do:

  1. perform Hyperparameter Tuning on the XGBoost model.
  2. select features using RFE.
  3. Validate result model on the testing set using cross-validation.

I've tried making some manipulations on my own, but ended up either having a bad performing model or data leakage.

How would you handle this type of problem? What sets should I balance and which not?

$\endgroup$
5
  • 1
    $\begingroup$ Artificially balancing the data doesn't solve a real problem. stats.stackexchange.com/questions/285231/… $\endgroup$
    – Sycorax
    Jul 17, 2023 at 13:22
  • 1
    $\begingroup$ Welcome to Cross Validated! Why do you want to balance at all? $\endgroup$
    – Dave
    Jul 17, 2023 at 13:22
  • 2
    $\begingroup$ Unless identifying relevant features is a specific goal of the analysis, you are probably better off not performing feature selection, but just to use regularisation to avoid overfitting. It is very easy to overfit the feature selection criterion. See stats.stackexchange.com/questions/27750/… $\endgroup$ Jul 17, 2023 at 13:24
  • 2
    $\begingroup$ And even if identifying the important features is a major goal, it is not a given that this can be done reliably! // Still, why balance the data at all? $\endgroup$
    – Dave
    Jul 17, 2023 at 13:27
  • $\begingroup$ is this a real project or just a student project? If it's a real project then log loss would be a better criterion, and looking at the probability estimate output (eg for calibration). accurate probabilities are necessary to calculate expected loss and other risk metrics in order to identify correct pricing $\endgroup$
    – seanv507
    Jul 17, 2023 at 14:22

1 Answer 1

3
$\begingroup$

Don't balance the dataset. As you are interested in "predicting credit card approvals" you have a task where different types of error have different costs. Giving someone a credit card when you shouldn't (because they have a high risk of defaulting on their debt) is likely to have a higher cost than not giving someone a credit card when you should (as the profit per transaction is likely to be fairly low). Work out plausible values for those misclassification costs and build them into your model selection and performance evaluation criteria. If you do that, normal modelling procedures ought to work fine.

In short, you need to work out what is really important in practice in the application (in the real world) and use that to work out what you need to do. When you evaluate the model make sure your criterion is measuring what is really important (in this case, it isn't accuracy - it is how much profit the credit provider is making).

$\endgroup$
8
  • 3
    $\begingroup$ Something interesting about this is that, if the data concern whether or not someone was approved (this seems to be the case), then the predictive model will only predict if someone will be approved, not if they should be approved. $\endgroup$
    – Dave
    Jul 17, 2023 at 13:32
  • 3
    $\begingroup$ ... plus the problem of likely having biased training data - if an applicant looked iffy in whatever system was used before to decide on approvals, they likely did not make it into the training data in the first place. $\endgroup$ Jul 17, 2023 at 13:35
  • 1
    $\begingroup$ @Dave and Spehan excellent points! - thinking about what we are really trying to achieve is vital, worrying about technical issues is irrelevant if we aren't answering the right question! $\endgroup$ Jul 17, 2023 at 13:36
  • $\begingroup$ thanks yall, I still have a question: if I don't balance any of given sets, will my model overfocus on majority class and will give wrong hyperparameters, hence will lead to poor prediction on testing set? $\endgroup$
    – CraZyCoDer
    Jul 17, 2023 at 13:41
  • 1
    $\begingroup$ Note that even including the misclassification costs, it may be that the optimal solution is to assign all patterns to the majority class, that is just the correct answer to the question. See stats.stackexchange.com/questions/539638/… $\endgroup$ Jul 17, 2023 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.