I tried several ways of selecting predictors for a logistic regression in R. I used lasso logistic regression to get rid of irrelevant features, cutting their number from 60 to 24, then I used those 24 variables in my stepAIC logistic regression, after which I further cut 1 variable with p-value of approximately 0.1. What other feature selection methods I can or even should use? I tried to look for Anova correlation coefficient analysis, but I didn't find any examples for R. And I think I cannot use correlation heatmap in this situation since my output is categorical? I seen some instances recommending Lasso and StepAIC, and other instances criticising them, but I didn't find any definitive comprehensive alternative, which left me confused.
I asked the above yesterday, but it was closed, so I provide further details below.
My sample is 300,000+ observations, so the number of variables isn't really the issue. The issue is their weak prediction power. The binary classification of the default/fully-paid loan classes I am doing has class imbalance of about 1:7, although it is not severe, it shows with PR-AUC scores of logreg, rf and xgboost models all under 30%. The accuracy with standard 0.5 threshold is 80%+ for these models, but such measures are irrelevant in classifications like this, where failing to correctly predict a default loan comes with high penalty. I know I have to try cost-sensitive modelling instead for better results, however I still need to show the low performance of standard approach ML models for my research. But a model like this logreg is giving only 60% sensitivity and 50% specifictiy after thresholding and about 65% accuracy at best, so I wanted to at least increase it's performance by conducting a proper feature selection procedure. And after all, I am still going to use this logistic regression IV set for the cost sensitive one, where it will basically just have weighted classes.
I first asked it on StackOverflow hoping that someone will share their usual feature selection procedure for logistic regression with code, but I was redirected here due to "more methodological nature" of my question, which is true ultimately; however, I would still highly appreciate links to code, examples of relevant feature selection methods, if anyone kindly shares them with me. Otherwise, I still would be happy to receive any purely theoretical advice.