Feature selection for Logistic Regression

Question

I am running a churn prediction model for an online ecommerce company. Since, volume of the data is high. I have historical data of around (~1M customers). On basis of market understanding I have selected 12 continuous variables as features. As a first step of logistic regression I have to do feature selection of which all features should be considered in logistic regression.

I am doing so by running logistic regressions keeping only 1 feature (Hence, running 12 logistic regressions). With the objective that I will select features which has p-value < 0.05. However, for all the 12 features I am getting p-value < 0.00001 hence suggesting that each of the variable is important, which I thought is highly unlikely. I reran the regression with randomly selected 0.1M data points even then I am witnessing same pattern.

My questions is this right approach to do feature selection when data volume is high?

Answer 1

My questions is this right approach to do feature selection when data volume is high?

Simply, no.

Basing feature selection on p values is a bad idea, especially when data are large. First, p-values tell you nothing about the effect of the variable. I can always construct a model with a highly significant feature but which performs negligibly different with respect to any classification metric you choose. This is because significant effects can be extremely small.

When data is large, the null is essentially a straw man. You have so much data that you can detect small effects because you have immense power to do so. The effect of any variable is never exactly 0 and you are finding that.

My advice is to use some principled modelling approach. People seem to like AIC (I'm not one of them), you could do forward feature selection (again, not my cup of tea), you could do lasso or ridge regression (I'm more keen on this), or frankly you could do none of them (my preference from what you've said in your post). If you have 12 variables which you know to be important, why aren't you using all of them? That's a rhetorical question.

In short, inference breaks down when you have so much data. The null becomes a straw man, so you reject near everything. People's obsession with p values leads to them using p values for things which they were not intended for (model selection). You should lean on methods which evaluate what you care about via a validation set or lean on your business knowledge.

EDIT:

I claim I can always make a model which performs negligibly better even when the p value is significant. Here is an example using linear regression:

library(tidyverse)
library(Metrics)

set.seed(0)

X = rnorm(1000000)
Z = rnorm(1000000)
y = 2*X + 0.01*Z + rnorm(1000000, 0, 0.3)

d = tibble(X = X, Z = Z, y = y, set = 
             sample(c('test','train'), replace = T, 
                    size = 1000000))
test = filter(d, set=='test')
train = filter(d, set=='train')

model1 = lm(y~X + Z, data = train)
model2 = lm(y~X, data= train)

rmse(test$y, predict(model1, newdata = test))
#> [1] 0.2996978
rmse(test$y, predict(model2, newdata = test))
#> [1] 0.2998523

^{Created on 2022-01-06 by the reprex package (v2.0.1)} The rmse for both models agrees up to 3 decimal places. That is good for all intents and purposes in my opinion. Note that the coefficient for Z is highly significant (it gives the smallest p value R can give). The combination of tiny effect size and massive sample is what causes this phenomena.

Answer 2

From computational perspective, 1M data points and 12 features for logistic regression is nothing, i.e., the computer can return results in seconds.

try this example in R, and you will see how fast we can fit.

d=data.frame(matrix(runif(1e6*12),ncol=12))
d$y=sample(c(0,1),1e6, replace = T)
fit = glm(y~.,d,family='binomial')

So if your concern is the computation. It is not necessary to do the feature selection.

---

On the other and, if you do feature selection, in most cases, the performance (classification accuracy) will be worse. This is because, intuitively, more information does not hurt, even the feature is completely irrelevant to the label, the algorithm will just set the coefficient to zero.

If your focus is classification accuracy instead of interpretability, I would use logistic regression with regularization. See my another answer for details

Regularization methods for logistic regression

Note that "stepwise regression, is now considered a statistical sin."

See this post

What are modern, easily used alternatives to stepwise regression?

Answer 3

I agree with the others that p values are not useful here and that regularized regression (ridge, elastic net, lasso) are potential way to go (elastic net might be more useful if the variables are correlated - but which one is best is an empirical question).

I would also decide whether theoretical or potential interactions in the predictors or nonlinearities in the relationships between the predictor and outcome are important to you. If so you will either need to create them ahead of time - here is a resource with potential considerations looking at interactions in a regularized regression. Also, if interested in interactions or nonlinear relationships you could consider using or combining your model with a random forest model. One popular option that I have also found success with is Boruta, which is a wrapper around a random forest model that examines whether your features are better than randomly permuted versions of the features. As Demetri pointed out above, any predictor with your sample size would likely have some nonzero relationship with the outcome, making p values for that purpose not useful. Yet, comparing whether the features are significantly better than their random permutations as a Boruta does is a way that a significant difference using p values can become useful again.

Either way, if the 12 variables you have are considered theoretically useful, you seem to have three options - keep them all (that's not a lot of features - why not include them all), trying to figure out if some can be dropped with two large a loss in prediction accuracy), or trying to figure out what relationships between these predictors and the outcome is the most useful for prediction. The second option seems to be what you are asking and might be fastest, but the third option might help you the most with prediction over time.

Answer 4

As with any regression it is best to either be well versed in the subject matter or work with a Subject Matter Expert (SME) to help determine which variables make sense.

A significant step in the process is to look at the stepwise results and see when the point of diminishing returns is reached. In other words, look at the amount of variance explained at each step. At some point the variance explained with significantly diminish which should help you determine a stopping point. Of course, you should always look at the correlation between predictors and how coefficients change as new variables are added and of course consult with the SME to determine which variables make the most sense.

Also, I would always recommend consulting with an experienced modeler and have them review the final product and the steps along the way, including any initial variable cleansing and transformations. BTW, I always recommend binning be considered with logistic regression.

Other factors are how easy are the variables to monitor and implement, which should always be a practical consideration.

BTW, I noticed the reference to the article on "Problems with stepwise.." and it is certainly valid; however, stepwise, used judiciously can yield effective, useful results. As with any modeling technique, the results should be tested on an independent (preferably out of time, if applicable) sample to validate the results.

Answer 5

Do you have a stepwise option on your Logistic Regression? That would be preferred.

While all 12 features may yield a significant p-value individually, they may not all be significant when considered in combination with one or more other features. You need to find the best subset.

In any case, it is not the p-values you want to be comparing. If you have significant p-values, what you want to compare is the proportion of variance accounted for. Choose the feature accounting for the largest proportion of variance. Once that is found, run 11 2-feature regressions using that first selected feature combined with each of the remaining 11 features in turn. Then pick the feature that accounts for the most additional variance (as long as the additional amount still has a significant p-value). That gives you the 2 best features. Continue with additional ones until you can no longer account for a significant amount of additional variance.

Obviously, this is a lot of work! But a step-wise option using all 12 variables will do all of this for you automatically. Sometimes there is also a "best subset" option that will effectively test all possible combinations of features to arrive at the best subset. This may not always give the same result as the stepwise option.