perfect separation logistic regression [duplicate]

Question

in continuity to the post stepwise logistic regression non significative variables(high p-values) and as demanded by matthew this is a post explaining the data i have and the problem in fact i have a data composed of 46274 values and 68 variables one y=(0=normal cell/1=abnormalcell) and 67 quantitative variables what i want is doing logistic regression y~ (other variables) but i have the following warnings

gg=glm(y~.,data=datf,family=binomial())
Warning messages:
1: glm.fit: algorithm did not converge 
2: glm.fit: fitted probabilities numerically 0 or 1 occurred

and what surprises me is that all variables are significant according to summary in addition the accuracy of confusion matrix is equal to 0.99

Call:
glm(formula = y ~ ., family = binomial(), data = datf)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
 -8.49    0.00    0.00    0.00    8.49  

Coefficients:
              Estimate Std. Error    z value Pr(>|z|)   
(Intercept) -1.795e+16  4.304e+07 -417170033   <2e-16 ***
a1          -3.220e+13  6.216e+05  -51803034   <2e-16 ***
a2           9.065e+13  5.924e+05  153010830   <2e-16 ***
a3          -1.767e+13  5.232e+05  -33763208   <2e-16 ***
p1          -1.223e+13  1.758e+05  -69542630   <2e-16 ***
p2          -9.098e+12  1.752e+05  -51928139   <2e-16 ***
p3           8.917e+12  1.564e+05   57009154   <2e-16 ***
cont         1.867e+14  1.030e+07   18118056   <2e-16 ***
eg           6.280e+14  7.219e+06   86983250   <2e-16 ***
h            5.468e+15  3.476e+07  157292659   <2e-16 ***
c1           6.532e+15  9.454e+07   69089908   <2e-16 ***
c2          -1.284e+14  3.977e+05 -322837213   <2e-16 ***
c3          -5.017e+13  1.775e+05 -282705283   <2e-16 ***
c4          -4.230e+13  8.652e+04 -488886033   <2e-16 ***
....

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 64149.4  on 46273  degrees of freedom
Residual deviance:  7497.1  on 46205  degrees of freedom
AIC: 7635.1

then after doing stepwise regression all the variables(33 variables) of selected model were non significatives the warning messages persist and accuracy was equal to 1

               Estimate   Std. Error  z value   Pr(>|z|)
(Intercept)    3.945e+04  3.178e+04   1.241     0.215    
a1            -2.499e+02  1.938e+02  -1.289     0.197    
p3             2.908e+01  2.391e+01   1.216     0.224
cont           1.279e+04  1.016e+04   1.259     0.208    
h              5.125e+04  3.948e+04   1.298     0.194    
c1            -2.490e+05  1.871e+05  -1.331     0.183    
c2            -3.996e+02  2.989e+02  -1.337     0.181
c7            -5.221e+01  3.932e+01  -1.328     0.184
....

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 6.4149e+04  on 46273  degrees of freedom
Residual deviance: 2.4470e-02  on 46240  degrees of freedom
AIC: 68.024
Number of Fisher Scoring iterations: 25

what conclusion should i put or what correction should i do thanks a lot in advance for any help

Answer 1

First, you should not use stepwise. That's been discussed here (and elsewhere) many times. The results are mostly incorrect.

Second, it's possible to have separation even when you have a lot of data. You could start by looking at how each independent variable relates to the dependent variable. Look at crosstabs for categorical IVs and perhaps parallel box plots for continuous IVs. You should also look at collinearity.

Third, the solution will probably be to run a model with fewer IVs.

Here is an example with N = 2000 and only one IV and perfect separation:

set.seed(1234)   #sets a seed

x <- c(rnorm(1000), rnorm(1000,5,0.1))  #Very separate, just for illustration
y <- c(rep(1,1000), rep(0,1000))  

m1 <- glm(y~x, family = binomial)