What feature selection methods to implement for logistic regression in R?

Question

I tried several ways of selecting predictors for a logistic regression in R. I used lasso logistic regression to get rid of irrelevant features, cutting their number from 60 to 24, then I used those 24 variables in my stepAIC logistic regression, after which I further cut 1 variable with p-value of approximately 0.1. What other feature selection methods I can or even should use? I tried to look for Anova correlation coefficient analysis, but I didn't find any examples for R. And I think I cannot use correlation heatmap in this situation since my output is categorical? I seen some instances recommending Lasso and StepAIC, and other instances criticising them, but I didn't find any definitive comprehensive alternative, which left me confused.

I asked the above yesterday, but it was closed, so I provide further details below.

My sample is 300,000+ observations, so the number of variables isn't really the issue. The issue is their weak prediction power. The binary classification of the default/fully-paid loan classes I am doing has class imbalance of about 1:7, although it is not severe, it shows with PR-AUC scores of logreg, rf and xgboost models all under 30%. The accuracy with standard 0.5 threshold is 80%+ for these models, but such measures are irrelevant in classifications like this, where failing to correctly predict a default loan comes with high penalty. I know I have to try cost-sensitive modelling instead for better results, however I still need to show the low performance of standard approach ML models for my research. But a model like this logreg is giving only 60% sensitivity and 50% specifictiy after thresholding and about 65% accuracy at best, so I wanted to at least increase it's performance by conducting a proper feature selection procedure. And after all, I am still going to use this logistic regression IV set for the cost sensitive one, where it will basically just have weighted classes.

I first asked it on StackOverflow hoping that someone will share their usual feature selection procedure for logistic regression with code, but I was redirected here due to "more methodological nature" of my question, which is true ultimately; however, I would still highly appreciate links to code, examples of relevant feature selection methods, if anyone kindly shares them with me. Otherwise, I still would be happy to receive any purely theoretical advice.

Answer 1

where failing to correctly predict a default loan comes with high penalty.

An astute observation. A good question might be "when is the loan worth risking the high penalty?".

You should not be classifying observations as default/not default in this scenario, but rather modelling the risk that the customer will not pay back the loan. With an appropriate risk model, you can then use a utility function to make decisions about if future loans should be awarded, rationally answering the question of when the risk of default is worth the possible reward. Frank Harrell writes more on the forced choice classification puts upon us and how treating problems probabilistically is a better approach. I suggest you read this post prior to continuing.

Additionally, I find that you really shouldn't be selecting features, you should be engineering new ones. If all you're doing is looking at linear effects, you're likely missing out on additional model performance. Non-linear models like random forests may be able to learn non-linear effects, however you still need to make sure the resulting fit is well calibrated before using it.

All in all, I think your approach needs to be revisited. It sounds like you're modelling loan defaults, which is very well studied and is best approached as a risk modelling problem and not a classification problem.