1
$\begingroup$

I have the following question: Can the response variable ever be used in a clustering algorithm?

I understand that in general, clustering is considered to be an "unsupervised learning algorithm" which is meant to be used when the response variable is not present. But in terms of statistical methodology, is there anything wrong with running a clustering algorithm (e.g. K-Means) on some data that has a response variable? In theory - could it even be considered more beneficial to run clustering on data that has a response variable, seeing as you now have more information about the data?

I illustrate this example below (using the R programming language). I generated the following data (where "age" is considered the response variable, and "height" and "weight" are the covariates):

set.seed(123)
height = rnorm(1000, 6,0.5)
weight = rnorm(1000, 200, 5)
age = rnorm(1000,30,1)
my_data1 = data.frame(height, weight, age)

height = rnorm(1000, 5.5,0.5)
weight = rnorm(1000, 150, 5)
age = rnorm(1000,20,1)
my_data2 = data.frame(height, weight, age)

final_data = rbind(my_data1, my_data2)

head(final_data)
    height   weight      age
1 5.719762 195.0210 29.48840
2 5.884911 194.8002 30.23694
3 6.779354 199.9101 29.45841
4 6.035254 199.3391 31.21923
5 6.064644 187.2533 30.17414
6 6.857532 205.2029 29.38473

Next, I run clustering algorithms with and without the response variable:

1) Without the Response Variable

library(ggplot2)

cls <- kmeans(x = final_data[, 1:2], centers = 2)

final_data$cluster = as.factor(cls$cluster)

ggplot() +
  geom_point(data = final_data, 
             mapping = aes(x = height, 
                                  y = weight, 
                                  colour = cluster)) + ggtitle("Clustering Without the Response Variable")

enter image description here

2) Clustering With the Response Variable

library(plotly)
library(dplyr)

  cls <- kmeans(x = final_data, centers = 2)
    
    final_data$cluster = as.factor(cls$cluster)

fig <- plot_ly(final_data, x = ~height, y = ~weight, z = ~age, color = ~cluster, colors = c('#BF382A', '#0C4B8E'))
fig <- fig %>% add_markers()
fig <- fig %>% layout(scene = list(xaxis = list(title = 'Height'),
                     yaxis = list(title = 'Weight'),
                     zaxis = list(title = 'Age')))

fig %>% layout(title = 'Clustering With the Response Variable ')

enter image description here

Question: In this example, the clustering performed on the artificially generated data does not seem to be affected by using the response variable compared to omitting the response variable. But in general, if the response variable is available - is there anything fundamentally wrong with using the response variable in a clustering algorithm? Could there be any benefits, seeing as you have additional information at your disposal?

Thanks!

$\endgroup$
4
  • 9
    $\begingroup$ Cluster analysis, as itself, does not know anything about "response" and "predictory" variables. It is outside of this distinction. Therefore the question to include or not to include a response into a clustering is solved on a higher level, the level of the project. The main question here is whether the inclusion of response brings about a contamination that is, adding of information that should be kept away for a while for the subsequent analysis. $\endgroup$
    – ttnphns
    Nov 4, 2021 at 8:33
  • 1
    $\begingroup$ that is a great point! thank you for your bringing this up! $\endgroup$
    – stats_noob
    Nov 5, 2021 at 0:38
  • $\begingroup$ @ttnphns - you might want to consider expanding that (mildly) into an answer... $\endgroup$
    – jbowman
    Apr 29, 2022 at 15:00
  • $\begingroup$ Your code is the same for both cases except the plotting. $\endgroup$
    – Tim
    May 1, 2022 at 21:29

1 Answer 1

1
$\begingroup$

The first thing to mention is that the two clusters in your data are very easily separated. This makes it hard to notice differences between approaches to improve the separation between clusters.

You can run the code below and see that the clusters will differ when taking the third variable in consideration. C here is caused by A and B so you can literally consider it a response variable.

library(ggplot2)
set.seed(123)
a = rnorm(1000)
b = rnorm(1000)
c = a+b + rnorm(1000)
df = data.frame(a, b, c)

cls <- kmeans(x = df[, 1:2], centers = 2)
df$cluster = as.factor(cls$cluster)
ggplot() +
  geom_point(data = df, 
             mapping = aes(x = a, 
                           y = b, 
                           colour = cluster)) + ggtitle("Clustering Without the Response Variable")
cls <- kmeans(x = df[, c('a', 'b', 'c')], centers = 2)
df$cluster = as.factor(cls$cluster)
ggplot() +
  geom_point(data = df, 
             mapping = aes(x = a, 
                           y = b, 
                           colour = cluster)) + ggtitle("Clustering With the Response Variable")

Without the response variable With the response variable

I understand that in general, clustering is considered to be an "unsupervised learning algorithm" which is meant to be used when the response variable is not present. But in terms of statistical methodology, is there anything wrong with running a clustering algorithm (e.g. K-Means) on some data that has a response variable?

There is nothing really preventing you from using approaches more commonly used in supervised learning to unsupervised learning problems and vice-versa (or semi-supervised, and so on). People do this all the time. Sometimes it works well, sometimes it doesn't. Take Support Vector Machines (SVM), for example. People usually think of it as a way to do classification in supervised learning. However, some other people have been using it for regression. Some other people for clustering in unsupervised learning. Same thing for Neural Networks (supervised, semi-supervised, unsupervised, and so on).

As for including a variable that in a supervised learning context would be seen as a response variable, it's fine, as long as you don't run into data leakage, what @ttnphns mentioned (contamination). Let's say you're trying to identify fraudulent purchases. When a new putative fraudulent purchase comes for analysis, do you have for that purchase all the variables used to separate the clusters before? If so, it's fine.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.