I am running a churn prediction model for an online ecommerce company. Since, volume of the data is high. I have historical data of around (~1M customers). On basis of market understanding I have selected 12 continous variables as features. AS a first step of logistics regression I have to do feature selection of which all features should be considered in logistic regression.

I am doing so by running logistics regression keeping only 1 feature (Hence, running 12 logistics regression). With the objective that I will select features which has p-value < 0.05. However, for all the 12 features I am getting p-value < 0.00001 hence suggesting that each of the variable is important, which I thought is highly unlikely. I reran the regression with randomly selected 0.1M data points even then I am witnessing same pattern.

My questions is this right approach to do feature selection when data volume is high?


My questions is this right approach to do feature selection when data volume is high?

Simply, no.

Basing feature selection on p values is a bad idea, especially when data are large. First, p-values tell you nothing about the effect of the variable. I can always construct a model with a highly significant feature but which performs negligibly different with respect to any classification metric you choose. This is because significant effects can be extremely small.

When data is large, the null is essentially a straw man. You have so much data that you can detect small effects because you have immense power to do so. The effect of any variable is never exactly 0 and you are finding that.

My advice is to use some principled modelling approach. People seem to like AIC (I'm not one of them), you could do forward feature selection (again, not my cup of tea), you could do lasso or ridge regression (I'm more keen on this), or frankly you could do none of them (my preference from what you've said in your post). If you have 12 variables which you know to be important, why aren't you using all of them? That's a rhetorical question.

In short, inference breaks down when you have so much data. The null becomes a straw man, so you reject near everything. People's obsession with p values leads to them using p values for things which they were not intended for (model selection). You should lean on methods which evaluate what you care about via a validation set or lean on your business knowledge.


I claim I can always make a model which performs negligibly better even when the p value is significant. Here is an example using linear regression:



X = rnorm(1000000)
Z = rnorm(1000000)
y = 2*X + 0.01*Z + rnorm(1000000, 0, 0.3)

d = tibble(X = X, Z = Z, y = y, set = sample(c('test','train'), replace = T, size = 1000000))
test = filter(d, set=='test')
train = filter(d, set=='train')

model1 = lm(y~X + Z, data = train)
model2 = lm(y~X, data= train)

rmse(test$y, predict(model1, newdata = test))
rmse(test$y, predict(model2, newdata = test))

The rmse for both models agrees up to 3 decimal places. That is good for all intents and purposes in my opinion. Note that the coefficient for Z is highly significant (it gives the smallest p value R can give). The combination of tiny effect size and massive sample is what causes this phenomena.

  • $\begingroup$ +1 for the nice answer. to make it even better, can you give an example of "I can always construct a model with a highly significant feature but which performs negligibly"? $\endgroup$ – Haitao Du Feb 27 '20 at 5:50
  • $\begingroup$ @HaitaoDu Done. $\endgroup$ – Demetri Pananos Feb 27 '20 at 5:59
  • $\begingroup$ Great, thank you very much. Always can learn from other people's code. This code also demonstrates the idea of my answer. If the target is improving accuracy, adding features will not hurt. $\endgroup$ – Haitao Du Feb 27 '20 at 6:43
  • $\begingroup$ @Dataist If you found this helpful, please upvote and accept the answer. $\endgroup$ – Demetri Pananos Feb 28 '20 at 17:05
  • $\begingroup$ Thanks it clarifies the concept to a great extend. I also understand that just relying on p-value is not the right thing and its better to look at how much variance is explained by each parameter. I gathered some information and learnt that in logistics, how much variance is explained is measured through Deviance as -2*(log likelihood of model - log likelihood of null model). Following are the values for each of the variable. Can I interpret as v1,v2,v3 and v4 don't explain much of the variance and hence are not important ones. Order of LLR 1,2,4,70,1054,1105,1237,1361,1444,2017,2637&1976 $\endgroup$ – Dataist Feb 28 '20 at 17:10

From computational perspective, 1M data points and 12 features for logistic regression is nothing, i.e., the computer can return results in seconds.

try this example in R, and you will see how fast we can fit.

d$y=sample(c(0,1),1e6, replace = T)
fit = glm(y~.,d,family='binomial')

So if your concern is the computation. It is not necessary to do the feature selection.

On the other and, if you do feature selection, in most cases, the performance (classification accuracy) will be worse. This is because, intuitively, more information does not hurt, even the feature is completely irrelevant to the label, the algorithm will just set the coefficient to zero.

If your focus is classification accuracy instead of interpretability, I would use logistic regression with regularization. See my another answer for details

Regularization methods for logistic regression

Note that "stepwise regression, is now considered a statistical sin."

See this post

What are modern, easily used alternatives to stepwise regression?

  • 1
    $\begingroup$ Thanks for the explanation !! $\endgroup$ – Dataist Feb 28 '20 at 17:10

Do you have a stepwise option on your Logistic Regression? That would be preferred.

While all 12 features may yield a significant p-value individually, they may not all be significant when considered in combination with one or more other features. You need to find the best subset.

In any case, it is not the p-values you want to be comparing. If you have significant p-values, what you want to compare is the proportion of variance accounted for. Choose the feature accounting for the largest proportion of variance. Once that is found, run 11 2-feature regressions using that first selected feature combined with each of the remaining 11 features in turn. Then pick the feature that accounts for the most additional variance (as long as the additional amount still has a significant p-value). That gives you the 2 best features. Continue with additional ones until you can no longer account for a significant amount of additional variance.

Obviously, this is a lot of work! But a step-wise option using all 12 variables will do all of this for you automatically. Sometimes there is also a "best subset" option that will effectively test all possible combinations of features to arrive at the best subset. This may not always give the same result as the stepwise option.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.