How can a logistic regression model have predictors that are highly insignificant and still perform excellently in the test dataset?

Question

I have built a classification model using a highly imbalanced dataset to be found in the ROSE package of R called hacide, containing 1,000 observations of which only 2% are positive. My model performs well in the test dataset rendering the following statistics:

> pred <- prediction(predictions, test$cls)
> auc = as.numeric(performance(pred, "auc")@y.values)
> auc
[1] 0.8942857

Still all the predictors of the model are highly insignificant --i.e. have very high p values. See below:

Call:
glm(formula = cls ~ ., family = "binomial", data = train)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.210   0.000   0.000   0.000   1.815  

Coefficients: (1 not defined because of singularities)
             Estimate Std. Error z value Pr(>|z|)
(Intercept)   -63.029  19632.716  -0.003    0.997
x1             15.446     16.542   0.934    0.350
x2              6.708      7.812   0.859    0.390
x11            65.696   6520.412   0.010    0.992
x12            30.091     33.700   0.893    0.372
x13           -18.419   6047.451  -0.003    0.998
x14           -14.663  26855.669  -0.001    1.000
x21            37.062  19632.675   0.002    0.998
x22            16.121  19726.706   0.001    0.999
x23            -2.200  20687.068   0.000    1.000
x24                NA         NA      NA       NA
x3              7.791      7.824   0.996    0.319

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 196.078  on 999  degrees of freedom
Residual deviance:  18.571  on 989  degrees of freedom
AIC: 40.571

Number of Fisher Scoring iterations: 25

My question is how is this possible? Should not we pay attention to the p values?

I understand that multicollinearity can play a role in raising the p-values and especially some of them, still we seldom see all the predictors to lose significance. Furthermore the correlations among the predictors in the specific example are not uniformly high. See below:

            cls           x1          x2         x11         x12          x13         x14         x21         x22         x23          x24
cls  1.00000000 -0.228158414 -0.21808962  0.62877027  0.06408317 -0.381460173 -0.06281125  0.25772091  0.06499037 -0.11428571 -0.094413507
x1  -0.22815841  1.000000000  0.05692690 -0.40907207 -0.47059084  0.339122434  0.02663309 -0.03553939 -0.06495859  0.04962622  0.008853023
x2  -0.21808962  0.056926900  1.00000000 -0.10014611 -0.04608160  0.072822544  0.66253601 -0.61372208 -0.30719548 -0.13965796  0.761337056
x11  0.62877027 -0.409072066 -0.10014611  1.00000000 -0.03614192 -0.329514671 -0.03192955  0.10787016  0.05614028 -0.04113450 -0.063671465
x12  0.06408317 -0.470590845 -0.04608160 -0.03614192  1.00000000 -0.782043609 -0.01809118  0.04226619  0.05305775 -0.07043852  0.015990310
x13 -0.38146017  0.339122434  0.07282254 -0.32951467 -0.78204361  1.000000000  0.01591565 -0.08952730 -0.06137774  0.08829694  0.002594731
x14 -0.06281125  0.026633095  0.66253601 -0.03192955 -0.01809118  0.015915651  1.00000000 -0.16082080 -0.12788433 -0.43967877  0.665278251
x21  0.25772091 -0.035539391 -0.61372208  0.10787016  0.04226619 -0.089527297 -0.16082080  1.00000000 -0.10638700 -0.36576885 -0.241734637
x22  0.06499037 -0.064958593 -0.30719548  0.05614028  0.05305775 -0.061377745 -0.12788433 -0.10638700  1.00000000 -0.29085855 -0.192226834
x23 -0.11428571  0.049626217 -0.13965796 -0.04113450 -0.07043852  0.088296938 -0.43967877 -0.36576885 -0.29085855  1.00000000 -0.660894552
x24 -0.09441351  0.008853023  0.76133706 -0.06367146  0.01599031  0.002594731  0.66527825 -0.24173464 -0.19222683 -0.66089455  1.000000000
x3   0.51362162 -0.199469306 -0.84437482  0.43450107  0.17141845 -0.389151137 -0.54093332  0.54322307  0.30336002  0.06389858 -0.627306333
             x3
cls  0.51362162
x1  -0.19946931
x2  -0.84437482
x11  0.43450107
x12  0.17141845
x13 -0.38915114
x14 -0.54093332
x21  0.54322307
x22  0.30336002
x23  0.06389858
x24 -0.62730633
x3   1.00000000

To make my example reproducible I provide the relevant R code:

library(GGally)
library(ggplot2)
library(data.table)
library(ROSE)
library(ROCR)
data(hacide)
train <- hacide.train
test <- hacide.test
train <- train %>% mutate(
  x11 = ifelse(x1 < -1.4, 1, 0),
  x12 = ifelse(((x1 >= -1.4) & (x1 < -0.74)), 1, 0),
  x13 = ifelse(((x1 >= -0.74) & (x1 < 1)), 1, 0),
  x14 = ifelse(x2 >= 1, 1, 0),
  x21 = ifelse(x2 < -1.4, 1, 0),
  x22 = ifelse(((x2 >= -1.4) & (x2 < -1)), 1, 0),
  x23 = ifelse(((x2 >= -1) & (x2 < 0.5)), 1, 0),
  x24 = ifelse(x2 >= 0.5, 1, 0),
  x3 = x1 ^ 2 - x2
)
test <- test %>% mutate(
  x11 = ifelse(x1 < -1.4, 1, 0),
  x12 = ifelse(((x1 >= -1.4) & (x1 < -0.74)), 1, 0),
  x13 = ifelse(((x1 >= -0.74) & (x1 < 1)), 1, 0),
  x14 = ifelse(x2 >= 1, 1, 0),
  x21 = ifelse(x2 < -1.4, 1, 0),
  x22 = ifelse(((x2 >= -1.4) & (x2 < -1)), 1, 0),
  x23 = ifelse(((x2 >= -1) & (x2 < 0.5)), 1, 0),
  x24 = ifelse(x2 >= 0.5, 1, 0),
  x3 = x1 ^ 2 - x2
)
pilot <- glm(cls ~ ., train, family = "binomial")
predictions <- predict(pilot, test, type = "response")
tbl <- table(test$cls , predictions > 0.5)
    FALSE TRUE
  0   244    1
  1     3    2

Answer 1

It could be multicollinearity, but if so that's just icing on the cake. Your basic problem is that your model is effectively saturated. In the context of linear models, a regression is saturated when the number of observations is equal to the number of variables plus one. For instance, if you have one X variable and only two data, you can perfectly fit the data with a straight line. (In fact, when there are fewer data than that, the model is unidentifiable; see: Fitting least squares when number of predictors are larger than instances.) In the OLS regression setting (again), we often say that a model is 'approaching saturation' when you have $<10$ data per variable.

Now in your case, you have a logistic (not OLS) regression model, and you have a thousand observations, so it would seem intuitive to assume that you must be OK. However, the corresponding rule of thumb for binary regression is that you need at least fifteen of the less commonly occurring outcome for every variable in order to avoid 'approaching saturation'. You say that you have only $2\%$ positive cases. Out of one thousand observations, that means you have only $20$ positive cases. Clearly positive is the less commonly occurring outcome here, so you want to have $15$ positive cases for each variable. But, you have $11$ variables! Thus, your model is 'approaching saturation' almost by definition (except that that isn't really a definition, it's an out-and-out hand-wavy rule of thumb). It is worth noting that the model could not even estimate one of the parameters; x24 has been rendered NA (cf., Maximum number of independent variables that can be entered into a multiple regression equation).

Nonetheless, you do have $20$ positive cases and 'only' $11$ variables, so based on what I said above, you might assume that you shouldn't really have complete saturation. And, in fact, you may not have complete saturation, but I'm guessing that your variables are categorical (viz., binary) and that there are instances in which you have more than one of the positive cases lying within a given cell, which reduces the effective number of positive cases via redundancy. Another way to think about that is that you almost certainly have perfect separation in your model, which again increases the saturation of the model. Indeed, you can see the classic hallmarks of separation: you have huge estimated coefficients, even more vastly huge standard errors associated with them, and a very large number of Fisher scoring iterations (namely, 25) were needed to try to fit the model. (To further understand separation in this context, it may help you to read my answers here: Adding interactions to logistic regression leads to high SEs and here: Logistic glm with good predictors is giving p-values = 1.)