Can the Response Variable ever be used in Clustering?

Question

I have the following question: Can the response variable ever be used in a clustering algorithm?

I understand that in general, clustering is considered to be an "unsupervised learning algorithm" which is meant to be used when the response variable is not present. But in terms of statistical methodology, is there anything wrong with running a clustering algorithm (e.g. K-Means) on some data that has a response variable? In theory - could it even be considered more beneficial to run clustering on data that has a response variable, seeing as you now have more information about the data?

I illustrate this example below (using the R programming language). I generated the following data (where "age" is considered the response variable, and "height" and "weight" are the covariates):

set.seed(123)
height = rnorm(1000, 6,0.5)
weight = rnorm(1000, 200, 5)
age = rnorm(1000,30,1)
my_data1 = data.frame(height, weight, age)

height = rnorm(1000, 5.5,0.5)
weight = rnorm(1000, 150, 5)
age = rnorm(1000,20,1)
my_data2 = data.frame(height, weight, age)

final_data = rbind(my_data1, my_data2)

head(final_data)
    height   weight      age
1 5.719762 195.0210 29.48840
2 5.884911 194.8002 30.23694
3 6.779354 199.9101 29.45841
4 6.035254 199.3391 31.21923
5 6.064644 187.2533 30.17414
6 6.857532 205.2029 29.38473

Next, I run clustering algorithms with and without the response variable:

1) Without the Response Variable

library(ggplot2)

cls <- kmeans(x = final_data[, 1:2], centers = 2)

final_data$cluster = as.factor(cls$cluster)

ggplot() +
  geom_point(data = final_data, 
             mapping = aes(x = height, 
                                  y = weight, 
                                  colour = cluster)) + ggtitle("Clustering Without the Response Variable")

2) Clustering With the Response Variable

library(plotly)
library(dplyr)

  cls <- kmeans(x = final_data, centers = 2)
    
    final_data$cluster = as.factor(cls$cluster)

fig <- plot_ly(final_data, x = ~height, y = ~weight, z = ~age, color = ~cluster, colors = c('#BF382A', '#0C4B8E'))
fig <- fig %>% add_markers()
fig <- fig %>% layout(scene = list(xaxis = list(title = 'Height'),
                     yaxis = list(title = 'Weight'),
                     zaxis = list(title = 'Age')))

fig %>% layout(title = 'Clustering With the Response Variable ')

Question: In this example, the clustering performed on the artificially generated data does not seem to be affected by using the response variable compared to omitting the response variable. But in general, if the response variable is available - is there anything fundamentally wrong with using the response variable in a clustering algorithm? Could there be any benefits, seeing as you have additional information at your disposal?

Thanks!

Answer 1

The first thing to mention is that the two clusters in your data are very easily separated. This makes it hard to notice differences between approaches to improve the separation between clusters.

You can run the code below and see that the clusters will differ when taking the third variable in consideration. C here is caused by A and B so you can literally consider it a response variable.

library(ggplot2)
set.seed(123)
a = rnorm(1000)
b = rnorm(1000)
c = a+b + rnorm(1000)
df = data.frame(a, b, c)

cls <- kmeans(x = df[, 1:2], centers = 2)
df$cluster = as.factor(cls$cluster)
ggplot() +
  geom_point(data = df, 
             mapping = aes(x = a, 
                           y = b, 
                           colour = cluster)) + ggtitle("Clustering Without the Response Variable")
cls <- kmeans(x = df[, c('a', 'b', 'c')], centers = 2)
df$cluster = as.factor(cls$cluster)
ggplot() +
  geom_point(data = df, 
             mapping = aes(x = a, 
                           y = b, 
                           colour = cluster)) + ggtitle("Clustering With the Response Variable")

I understand that in general, clustering is considered to be an "unsupervised learning algorithm" which is meant to be used when the response variable is not present. But in terms of statistical methodology, is there anything wrong with running a clustering algorithm (e.g. K-Means) on some data that has a response variable?

There is nothing really preventing you from using approaches more commonly used in supervised learning to unsupervised learning problems and vice-versa (or semi-supervised, and so on). People do this all the time. Sometimes it works well, sometimes it doesn't. Take Support Vector Machines (SVM), for example. People usually think of it as a way to do classification in supervised learning. However, some other people have been using it for regression. Some other people for clustering in unsupervised learning. Same thing for Neural Networks (supervised, semi-supervised, unsupervised, and so on).

As for including a variable that in a supervised learning context would be seen as a response variable, it's fine, as long as you don't run into data leakage, what @ttnphns mentioned (contamination). Let's say you're trying to identify fraudulent purchases. When a new putative fraudulent purchase comes for analysis, do you have for that purchase all the variables used to separate the clusters before? If so, it's fine.