3
$\begingroup$

I am trying to generate a decision boundary of logistic regression. My Training set is 2/3 and the test set is 1/3, I have however tried producing the decision boundary but not sure whether is it the desired behavior or not. I am using the caret library for the logistic model and the lattice library for the plot.

here is my approach:

#Data creation
library(mvtnorm)

a1 <- c(1, 0)
a2 <- c(0, 1)
M <- cbind(a1, a2)

C0 <- rmvnorm(100, c(0, 0), M)
C1 <- rmvnorm(100, c(5, 0), M)

dat <- rbind(C0, C1)
C <- data.frame(dat)
y <- sign(-1 - 2 * dat[,1] + 4 * dat[,2] )
y[y == -1] <- 0
df1 <- cbind.data.frame(y, C)
df1
 
library(caret)
#Create training and test sets
set.seed(123)
trainIndex <- sample(c(FALSE,TRUE), size = nrow(df1), prob = 
                     c(.33,.67), replace = TRUE)
train_set <- df1[trainIndex, ]
test_set <- df1[!trainIndex, ]
# Learn Logistic Regression Model
fit <- glm(y ~ ., data = train_set, family = "binomial")
pred <- predict(fit, newdata = test_set, type = "response")
tab <- table(actual = test_set$y, predicted = round(pred))
cm1 <- confusionMatrix(tab)
cm1
slope <- coef(fit)[2]/(-coef(fit)[3])
intercept <- coef(fit)[1]/(-coef(fit)[3]) 

library(lattice)
xyplot( x2 ~ x1  , data = df1, groups = y,
   panel=function(...){
       panel.xyplot(...)
       panel.abline(intercept , slope)
       panel.grid(...)
       })

My decision boundary

decision boundary

As you can observe it has not got any clear separation of two classes which I'm highly uncertain of.

Also, I have learned that the hypothesis space is represented as a set of the conjunction of constraints but I cannot visualize this in R. Is it the decision boundary per se or there are different plots/visualizations to represent it?

$\endgroup$
7
  • $\begingroup$ The data set looks small for a classification problem. Also does not look like the classes (blue, magenta) are linearly separable. Are X1 and X2 the only features? Have you looked at kNN? $\endgroup$
    – user318288
    Apr 20, 2021 at 21:31
  • $\begingroup$ @nxglogic, the dataset is just for a simulation purpose for a baseline check with logit model, Yes, X1 and X2 are the features. $\endgroup$
    – Ranji Raj
    Apr 20, 2021 at 21:39
  • $\begingroup$ ok, increase the sample size, and consider k-fold CV, since 2/3 training and 1/3 is inefficient. See my answer on k-fold CV ( stats.stackexchange.com/questions/519698/… ). $\endgroup$
    – user318288
    Apr 20, 2021 at 21:44
  • $\begingroup$ Thanks for the redirection. I'll check that out! $\endgroup$
    – Ranji Raj
    Apr 20, 2021 at 21:46
  • $\begingroup$ how did you come up with your slope and intercept? $\endgroup$
    – rep_ho
    Apr 20, 2021 at 21:46

1 Answer 1

2
$\begingroup$

A couple of comments...

First, the data are completely separable. There is a hyper plane in $(x_1, x_2)$ space which can completely separate the positive and negative case. This is bad and results in a logistic regression which does not converge

 model = glm(y~., data = train_set, family = binomial())
Warning messages:
1: glm.fit: algorithm did not converge 
2: glm.fit: fitted probabilities numerically 0 or 1 occurred 

If you're looking for a better way to generate data, might I suggest this thread.

None the less, you can use the model and plot the decision boundary. The decision boundary is where

$$ 0 = \beta_0 + \beta_1 x_1 + \beta_2 x_2 $$

In $(x_1, x_2)$ space, that would be

$$ x_2 = -\dfrac{\beta_0}{\beta_2} - \dfrac{\beta_1}{\beta_2}x_1$$

As you've correctly identified. Let's plot the training data and this line

b = coef(model)
slope = -b[2]/b[3]
int = -b[1]/b[3]

train_set %>% 
  ggplot(aes(X1, X2, color = y))+
  geom_point()+
  geom_abline(aes(slope = slope, intercept=int), color = 'red')

enter image description here

The plane seperates the two classes perfectly (which is why glm throws a complaint). Plotting the decision boundary and the test set results in a similar picture; the two classes are perfectly separated.

So, aside from generating data which can not be fit by logistic regression, you've done everything perfectly. I'm not sure why you're experiencing a problem. You might have an error in your plotting code (I use ggplot2, not lattice).

$\endgroup$
3
  • 1
    $\begingroup$ You say: ` This is bad and results in a logistic regression which does not converge` Why bad? While the coefficient estimates don't converge to finite values, the decision boundary converges ... and so long your goal is classification and not inference, that is very well, and a good outcome! See math.yorku.ca/Who/Faculty/Monette/S-news/0027.html and especially the B Ripley quote at stats.stackexchange.com/questions/415247/… $\endgroup$ Apr 20, 2021 at 23:18
  • 1
    $\begingroup$ @kjetilbhalvorsen The perspective on if a failure to converge is bad or not is, as you've said, entirely dependent on context. OP seems to just be doing classification, but I err on the side of caution lest someone think that confidence intervals or p values are still valid when this happens. $\endgroup$ Apr 20, 2021 at 23:40
  • $\begingroup$ @DemetriPananos, Yeah, I pretty much now get the idea of which libraries to use with. Appreciate your time for this detailed explanations :) $\endgroup$
    – Ranji Raj
    Apr 21, 2021 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.