Decision boundary of logistic Regression and Hypothesis space in R

Question

I am trying to generate a decision boundary of logistic regression. My Training set is 2/3 and the test set is 1/3, I have however tried producing the decision boundary but not sure whether is it the desired behavior or not. I am using the caret library for the logistic model and the lattice library for the plot.

here is my approach:

#Data creation
library(mvtnorm)

a1 <- c(1, 0)
a2 <- c(0, 1)
M <- cbind(a1, a2)

C0 <- rmvnorm(100, c(0, 0), M)
C1 <- rmvnorm(100, c(5, 0), M)

dat <- rbind(C0, C1)
C <- data.frame(dat)
y <- sign(-1 - 2 * dat[,1] + 4 * dat[,2] )
y[y == -1] <- 0
df1 <- cbind.data.frame(y, C)
df1
 
library(caret)
#Create training and test sets
set.seed(123)
trainIndex <- sample(c(FALSE,TRUE), size = nrow(df1), prob = 
                     c(.33,.67), replace = TRUE)
train_set <- df1[trainIndex, ]
test_set <- df1[!trainIndex, ]
# Learn Logistic Regression Model
fit <- glm(y ~ ., data = train_set, family = "binomial")
pred <- predict(fit, newdata = test_set, type = "response")
tab <- table(actual = test_set$y, predicted = round(pred))
cm1 <- confusionMatrix(tab)
cm1
slope <- coef(fit)[2]/(-coef(fit)[3])
intercept <- coef(fit)[1]/(-coef(fit)[3]) 

library(lattice)
xyplot( x2 ~ x1  , data = df1, groups = y,
   panel=function(...){
       panel.xyplot(...)
       panel.abline(intercept , slope)
       panel.grid(...)
       })

My decision boundary

As you can observe it has not got any clear separation of two classes which I'm highly uncertain of.

Also, I have learned that the hypothesis space is represented as a set of the conjunction of constraints but I cannot visualize this in R. Is it the decision boundary per se or there are different plots/visualizations to represent it?

Answer 1

A couple of comments...

First, the data are completely separable. There is a hyper plane in $(x_1, x_2)$ space which can completely separate the positive and negative case. This is bad and results in a logistic regression which does not converge

 model = glm(y~., data = train_set, family = binomial())
Warning messages:
1: glm.fit: algorithm did not converge 
2: glm.fit: fitted probabilities numerically 0 or 1 occurred 

If you're looking for a better way to generate data, might I suggest this thread.

None the less, you can use the model and plot the decision boundary. The decision boundary is where

$$ 0 = \beta_0 + \beta_1 x_1 + \beta_2 x_2 $$

In $(x_1, x_2)$ space, that would be

$$ x_2 = -\dfrac{\beta_0}{\beta_2} - \dfrac{\beta_1}{\beta_2}x_1$$

As you've correctly identified. Let's plot the training data and this line

b = coef(model)
slope = -b[2]/b[3]
int = -b[1]/b[3]

train_set %>% 
  ggplot(aes(X1, X2, color = y))+
  geom_point()+
  geom_abline(aes(slope = slope, intercept=int), color = 'red')

The plane seperates the two classes perfectly (which is why glm throws a complaint). Plotting the decision boundary and the test set results in a similar picture; the two classes are perfectly separated.

So, aside from generating data which can not be fit by logistic regression, you've done everything perfectly. I'm not sure why you're experiencing a problem. You might have an error in your plotting code (I use ggplot2, not lattice).