Testing Logistic Regression Classifier in R

Question

I am testing the logistic regression classifier in R. I created some test data like this:

x=runif(10000)
y=runif(10000)
df=data.frame(x,y,as.factor(x-y>0))

basically I am sampling the 2D unit square [0,1] and classifying a point belonging to class A or B depending on which side of y=x it lies.

I generated a scatter plot of the data like below:

names(df) = c("feature1", "feature2", "class")
levels=levels(df[[3]])
obs1=as.matrix(subset(df,class==levels[[1]])[,1:2])
obs2=as.matrix(subset(df,class==levels[[2]])[,1:2])
# make scatter plot
dev.new()
plot(obs1[,1],obs1[,2],xlab="x",ylab="y",main="scatter plot",pch=0,col=colors[[1]])
points(obs2[,1],obs2[,2],xlab="x",ylab="y",main="scatter plot",pch=1,col=colors[[2]])

it gives me below graph:

Now I tried running LR (logistic regression) on this data using code below:

model=glm(class~.,family="binomial",data=df)
summary(model) # prints summary

here are the results:

Call:
glm(formula = class ~ ., family = "binomial", data = df)

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-0.11832   0.00000   0.00000   0.00000   0.08847  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept)  5.765e-01  1.923e+01   0.030    0.976
feature1     9.761e+04  8.981e+04   1.087    0.277
feature2    -9.761e+04  8.981e+04  -1.087    0.277

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1.3863e+04  on 9999  degrees of freedom
Residual deviance: 2.9418e-02  on 9997  degrees of freedom
AIC: 6.0294

Number of Fisher Scoring iterations: 25

I also get these warning messages:

Warning messages:
1: glm.fit: algorithm did not converge 
2: glm.fit: fitted probabilities numerically 0 or 1 occurred 

If I try plotting the ROC curve using a varying threshold, I get following graph (AUC=1 which is good):

Could someone please explain why the algorithm does not converge and coefficient estimates are not statistically significant (high std. error in coeff estimates)?

I also compared to LDA:

lda_classifier=lda(class~., data=df)

gives:

Call:
lda(class ~ ., data = df)

Prior probabilities of groups:
 FALSE   TRUE 
0.5007 0.4993 

Group means:
       feature1  feature2
FALSE 0.3346288 0.6676169
TRUE  0.6710111 0.3380432

Coefficients of linear discriminants:
               LD1
**feature1  4.280490
feature2 -4.196388**

Answer 1

As you note, the standard errors are huge and the estimation algorithm doesn't really converge. There's two related things going on.

1. you have a two-dimensional predictor (two features), but a single linear combination of them is essentially perfectly related to the response. The relationship as a function of that linear combination (x-y) is very strong. The orthogonal direction is not at all determined.

2. you have complete separation in that ($x-y$) linear combination; that causes unstable estimates, since the fit of the model can be made "better" by inflating the magnitude of the coefficients to make the logistic curve increase more and more sharply:

--

While the parameter estimates for the two features individually are neither stable nor well-determined, the model still classifies perfectly. Performance of the fitted model on some new data: