Logistic regression vs clustering analysis

Question

I am having some trouble understanding the difference between clustering and logistic regression. Can you give me some examples of when and why it would be better to use clustering instead of logistic regression and vice versa?

Answer 1

I would start by considering that logistic regression is a method, a model in fact, whereas clustering is a family of methods so you are not really comparing like with like.

In any case, logistic regression can be described as supervised. You start with a dataset where you know whether each observation is "0" or "1" and you have a number of predictor variables. You build a model that allows you to estimate the contribution of each predictor to the classification and/or predict the class of future observations given new values of the predictors.

Clustering, typically, is unsupervised. You have observations for which a number of predictor variables are known but you don't know the assignment of observations to classes. In fact, you may not even known if and how many classes you have. So you can use clustering to learn about if/how many classes are there.

A dataset for clustering may look like this. You have two variables for 20 observations:

     X1   X2
1  12.6 15.2
2  15.0 16.2
3  14.5 18.7
4  10.7 15.6
5   7.5 15.7
6   2.6 15.0
7  13.5 14.4
8   2.8 17.7
9  15.2 17.8
10  7.1 11.9
11 22.0 21.4
12 21.6  9.8
13 17.1  7.0
14 10.7 10.0
15 20.7 17.9
16 26.2  4.9
17 23.2 16.1
18 18.8 15.9
19 21.4 11.5
20 20.8 10.0

You apply, for example, kmeans clustering and you find that there seem to be two groups:

---

For logistic regression, the dataset may be like this (the same but with classes given):

   class   X1   X2
1      0 12.6 15.2
2      0 15.0 16.2
3      0 14.5 18.7
4      0 10.7 15.6
5      0  7.5 15.7
6      0  2.6 15.0
7      0 13.5 14.4
8      0  2.8 17.7
9      0 15.2 17.8
10     1  7.1 11.9
11     1 22.0 21.4
12     1 21.6  9.8
13     1 17.1  7.0
14     1 10.7 10.0
15     1 20.7 17.9
16     1 26.2  4.9
17     1 23.2 16.1
18     1 18.8 15.9
19     1 21.4 11.5
20     1 20.8 10.0

You can use this dataset to estimate the contribution of X1 and X2 in determining the class and in turn use the estimates to predict new observations:

fit <- glm(data= dat, class ~ X1 + X2)
summary(fit)
...
Coefficients:
        Estimate Std. Error z value Pr(>|z|)  
(Intercept)    4.317      4.896    0.88    0.378  
X1             0.655      0.358    1.83    0.067 .
X2            -0.886      0.531   -1.67    0.095 .
...

Predict new observations:

# More likely a "1"
predict(fit, data.frame(X1= 20, X2= 15), type= 'response')
    1 
0.984 
 
# More likely a "0"
predict(fit, data.frame(X1= 15, X2= 17), type= 'response')
    1 
0.286 

---

More manually, you can use the estimates from the fitted model ($\beta$) to make predictions on new values $\textbf{X}$ considering that the fitted values are probabilities transformed by the logit function [$logit(p) = log(\frac{p}{1-p})$]. So we need to invert the logit function to obtain a probability [$logit^{-1}(x) = 1/(1+e^{-x})$]. So the probability of being 1 is:

$$ P(1) = logit^{-1}(\beta X) $$

In R:

invlogit <- function(x) { 1/(1+exp(-x)) }

est <- summary(fit)$coefficients[,1]
x1 <- 20
x2 <- 15
invlogit(est %*% c(1, x1, x2))
[1,] 0.9842

---

Here's the R code to reproduce:

set.seed(1234)
class <- rep(c(0, 1), c(9, 11))
xx <- rbind(
    mvrnorm(n= 10, mu= c(10, 20), diag(20, 2)),
    mvrnorm(n= 10, mu= c(20, 10), diag(20, 2))
    )
dat <- data.frame(class, xx)
plot(dat$X1, dat$X2, col= dat$class+1, pch= 19)

km <- kmeans(dat[, c('X1', 'X2')], centers= 2)
km <- cbind(dat, cluster= km$cluster)
plot(km$X1, km$X2, col= km$cluster, pch= 19)

fit <- glm(data= dat, class ~ X1 + X2, family= 'binomial')
summary(fit)

predict(fit, data.frame(X1= 20, X2= 15), type= 'response')
predict(fit, data.frame(X1= 15, X2= 17), type= 'response')

Answer 2

Succinctly,

• Logistic regression is a supervised learning task. If you have inputs and a binary outcome then you can use logistic regression. For example, let's say I want to know the probability of a plant dying using the mass of herbicide I use as my predictor. The outcome is did the plant die (yes or no, hence binary) and the predictor would be the mass of the herbicide.

• Clustering is an unsupervised learning task. If all you have is data and no outcomes, clustering is one way to find observations which are similar to one another in some sense. For example, let's say I have behaviour data for customers of some store. I have how frequently they come to the store, how much money they spend, the proportion of time they spend in each part of the store, etc etc. I can use clustering to make claims like "Customer X and Y behave similarly (because my algorithm puts them in the same cluster), but Customer Z is not similar to either Customer X or Customer Y because Customer Z is in a different cluster."