I created some data using the following code:
set.seed(1221) x <- runif(500) y <- runif(500, 0, 2) z <- rep(0, 500) z[-0.8*x + y - 0.75 > 0] <- 1 plot(x, y, col=as.factor(z))
This produces the following plot
The data is linearly separable. Then, I applied the
glm function to create a logistic regression model.
df <- data.frame(class = z, x = x, y = y) model <- glm(z ~ x + y, family = binomial, data = df)
This produces the following output:
summary(model) Call: glm(formula = z ~ x + y, family = binomial, data = df) Deviance Residuals: Min 1Q Median 3Q Max -8.127e-04 -2.000e-08 -2.000e-08 2.000e-08 7.699e-04 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -1062 52666 -0.02 0.984 x -1163 57197 -0.02 0.984 y 1433 70408 0.02 0.984 (Dispersion parameter for binomial family taken to be 1) Null deviance: 6.8274e+02 on 499 degrees of freedom Residual deviance: 1.3345e-06 on 497 degrees of freedom AIC: 6 Number of Fisher Scoring iterations: 25
The result surprised me, first because the parameter estimates are huge, and second because I was expecting such estimates to be close to the original decision boundary function, i.e.
-0.8x + y - 0.75 = 0.
I then used the glmnet package to see if I could solve this issue. This package creates a penalised logistic regression model in order to deal with the large values in the parameter estimates. The code I used is the following:
library(glmnet) cvfit <- cv.glmnet(as.matrix(df[,-1]), as.factor(df$class), family = "binomial", type.measure = "class") plot(cvfit)
And the coefficients for the optimal penalty strength are:
coef(cvfit, s = "lambda.min") 3 x 1 sparse Matrix of class "dgCMatrix" 1 (Intercept) -84.01446 x -91.40983 y 113.18736
Such coefficients are smaller than the ones obtained with the
glm function. Still they are not the same as the decision boundary function.
Does anybody know why this is happening? Any help is greatly appreciated.