# How can a logistic regression model have predictors that are highly insignificant and still perform excellently in the test dataset?

I have built a classification model using a highly imbalanced dataset to be found in the ROSE package of R called hacide, containing 1,000 observations of which only 2% are positive. My model performs well in the test dataset rendering the following statistics:

> pred <- prediction(predictions, test$cls) > auc = as.numeric(performance(pred, "auc")@y.values) > auc [1] 0.8942857  Still all the predictors of the model are highly insignificant --i.e. have very high p values. See below: Call: glm(formula = cls ~ ., family = "binomial", data = train) Deviance Residuals: Min 1Q Median 3Q Max -1.210 0.000 0.000 0.000 1.815 Coefficients: (1 not defined because of singularities) Estimate Std. Error z value Pr(>|z|) (Intercept) -63.029 19632.716 -0.003 0.997 x1 15.446 16.542 0.934 0.350 x2 6.708 7.812 0.859 0.390 x11 65.696 6520.412 0.010 0.992 x12 30.091 33.700 0.893 0.372 x13 -18.419 6047.451 -0.003 0.998 x14 -14.663 26855.669 -0.001 1.000 x21 37.062 19632.675 0.002 0.998 x22 16.121 19726.706 0.001 0.999 x23 -2.200 20687.068 0.000 1.000 x24 NA NA NA NA x3 7.791 7.824 0.996 0.319 (Dispersion parameter for binomial family taken to be 1) Null deviance: 196.078 on 999 degrees of freedom Residual deviance: 18.571 on 989 degrees of freedom AIC: 40.571 Number of Fisher Scoring iterations: 25  My question is how is this possible? Should not we pay attention to the p values? I understand that multicollinearity can play a role in raising the p-values and especially some of them, still we seldom see all the predictors to lose significance. Furthermore the correlations among the predictors in the specific example are not uniformly high. See below:  cls x1 x2 x11 x12 x13 x14 x21 x22 x23 x24 cls 1.00000000 -0.228158414 -0.21808962 0.62877027 0.06408317 -0.381460173 -0.06281125 0.25772091 0.06499037 -0.11428571 -0.094413507 x1 -0.22815841 1.000000000 0.05692690 -0.40907207 -0.47059084 0.339122434 0.02663309 -0.03553939 -0.06495859 0.04962622 0.008853023 x2 -0.21808962 0.056926900 1.00000000 -0.10014611 -0.04608160 0.072822544 0.66253601 -0.61372208 -0.30719548 -0.13965796 0.761337056 x11 0.62877027 -0.409072066 -0.10014611 1.00000000 -0.03614192 -0.329514671 -0.03192955 0.10787016 0.05614028 -0.04113450 -0.063671465 x12 0.06408317 -0.470590845 -0.04608160 -0.03614192 1.00000000 -0.782043609 -0.01809118 0.04226619 0.05305775 -0.07043852 0.015990310 x13 -0.38146017 0.339122434 0.07282254 -0.32951467 -0.78204361 1.000000000 0.01591565 -0.08952730 -0.06137774 0.08829694 0.002594731 x14 -0.06281125 0.026633095 0.66253601 -0.03192955 -0.01809118 0.015915651 1.00000000 -0.16082080 -0.12788433 -0.43967877 0.665278251 x21 0.25772091 -0.035539391 -0.61372208 0.10787016 0.04226619 -0.089527297 -0.16082080 1.00000000 -0.10638700 -0.36576885 -0.241734637 x22 0.06499037 -0.064958593 -0.30719548 0.05614028 0.05305775 -0.061377745 -0.12788433 -0.10638700 1.00000000 -0.29085855 -0.192226834 x23 -0.11428571 0.049626217 -0.13965796 -0.04113450 -0.07043852 0.088296938 -0.43967877 -0.36576885 -0.29085855 1.00000000 -0.660894552 x24 -0.09441351 0.008853023 0.76133706 -0.06367146 0.01599031 0.002594731 0.66527825 -0.24173464 -0.19222683 -0.66089455 1.000000000 x3 0.51362162 -0.199469306 -0.84437482 0.43450107 0.17141845 -0.389151137 -0.54093332 0.54322307 0.30336002 0.06389858 -0.627306333 x3 cls 0.51362162 x1 -0.19946931 x2 -0.84437482 x11 0.43450107 x12 0.17141845 x13 -0.38915114 x14 -0.54093332 x21 0.54322307 x22 0.30336002 x23 0.06389858 x24 -0.62730633 x3 1.00000000  To make my example reproducible I provide the relevant R code: library(GGally) library(ggplot2) library(data.table) library(ROSE) library(ROCR) data(hacide) train <- hacide.train test <- hacide.test train <- train %>% mutate( x11 = ifelse(x1 < -1.4, 1, 0), x12 = ifelse(((x1 >= -1.4) & (x1 < -0.74)), 1, 0), x13 = ifelse(((x1 >= -0.74) & (x1 < 1)), 1, 0), x14 = ifelse(x2 >= 1, 1, 0), x21 = ifelse(x2 < -1.4, 1, 0), x22 = ifelse(((x2 >= -1.4) & (x2 < -1)), 1, 0), x23 = ifelse(((x2 >= -1) & (x2 < 0.5)), 1, 0), x24 = ifelse(x2 >= 0.5, 1, 0), x3 = x1 ^ 2 - x2 ) test <- test %>% mutate( x11 = ifelse(x1 < -1.4, 1, 0), x12 = ifelse(((x1 >= -1.4) & (x1 < -0.74)), 1, 0), x13 = ifelse(((x1 >= -0.74) & (x1 < 1)), 1, 0), x14 = ifelse(x2 >= 1, 1, 0), x21 = ifelse(x2 < -1.4, 1, 0), x22 = ifelse(((x2 >= -1.4) & (x2 < -1)), 1, 0), x23 = ifelse(((x2 >= -1) & (x2 < 0.5)), 1, 0), x24 = ifelse(x2 >= 0.5, 1, 0), x3 = x1 ^ 2 - x2 ) pilot <- glm(cls ~ ., train, family = "binomial") predictions <- predict(pilot, test, type = "response") tbl <- table(test$cls , predictions > 0.5)
FALSE TRUE
0   244    1
1     3    2

• Firstly, p < some threshold is not a good criterion for what terms should be a in model and whether a model will predict well. Secondly, your model main include terms that do not contribute much, but the overall model does a good job. Multicollinearity could play a role. – Björn Jul 13 '17 at 13:45
• if you predict for all observations a negative outcome, you will be right for 98% of your observations, so getting 89% right is actually pretty bad performance. – Maarten Buis Jul 13 '17 at 14:51
• You confuse AUC with Accuracy. – AlK Jul 13 '17 at 15:38
• This phenomenon pertains to regression generally, not just logistic regression. Although some amount of correlation among the regressors is implicated, the multicollinearity need not be great. See stats.stackexchange.com/questions/3549 for a discussion. – whuber Jul 13 '17 at 16:05
• Do you understand the different between sensitivity and specificity, between false positive error and false negative error, and between positive predictive value and negative predictive value? – Alexis Aug 4 '17 at 2:41

It could be multicollinearity, but if so that's just icing on the cake. Your basic problem is that your model is effectively saturated. In the context of linear models, a regression is saturated when the number of observations is equal to the number of variables plus one. For instance, if you have one X variable and only two data, you can perfectly fit the data with a straight line. (In fact, when there are fewer data than that, the model is unidentifiable; see: Fitting least squares when number of predictors are larger than instances.) In the OLS regression setting (again), we often say that a model is 'approaching saturation' when you have $<10$ data per variable.
Now in your case, you have a logistic (not OLS) regression model, and you have a thousand observations, so it would seem intuitive to assume that you must be OK. However, the corresponding rule of thumb for binary regression is that you need at least fifteen of the less commonly occurring outcome for every variable in order to avoid 'approaching saturation'. You say that you have only $2\%$ positive cases. Out of one thousand observations, that means you have only $20$ positive cases. Clearly positive is the less commonly occurring outcome here, so you want to have $15$ positive cases for each variable. But, you have $11$ variables! Thus, your model is 'approaching saturation' almost by definition (except that that isn't really a definition, it's an out-and-out hand-wavy rule of thumb). It is worth noting that the model could not even estimate one of the parameters; x24 has been rendered NA (cf., Maximum number of independent variables that can be entered into a multiple regression equation).
Nonetheless, you do have $20$ positive cases and 'only' $11$ variables, so based on what I said above, you might assume that you shouldn't really have complete saturation. And, in fact, you may not have complete saturation, but I'm guessing that your variables are categorical (viz., binary) and that there are instances in which you have more than one of the positive cases lying within a given cell, which reduces the effective number of positive cases via redundancy. Another way to think about that is that you almost certainly have perfect separation in your model, which again increases the saturation of the model. Indeed, you can see the classic hallmarks of separation: you have huge estimated coefficients, even more vastly huge standard errors associated with them, and a very large number of Fisher scoring iterations (namely, 25) were needed to try to fit the model. (To further understand separation in this context, it may help you to read my answers here: Adding interactions to logistic regression leads to high SEs and here: Logistic glm with good predictors is giving p-values = 1.)