I'd have 10 groups and hundreds of observations per group. In this toy example I only have 3 groups with 20 observations each.
I am looking to see if groups are similar so I'm using kmeans to cluster them using 3 variables (x1, x2 and x3).
My questions is, when you have multiple observations per group how do determine the cluster a group is in given that not all 20 observations will be in the same cluster.
AS an example, the below example using all the 20 rows of data for each group and passes it to k means. As you can see not all of the 20 points end up in the same cluster for each group. Some of group A's points are in cluster 1 and some and in cluster 2. The goal it put group A into a single cluster. How can this be done using the individual row level data? Are there weighting schemes or furhter clustering techniques? i.e. if 51% of group A's points are in cluster 3 then call it cluster 3? Are there any paper on this problem?
library(ggplot2) library(dplyr) ########### CLUSTERING AT THE ROW level...points for a group are NOT all in the same cluster. How to handle that? data = data.frame(group = rep(c("A","B","C"),each =20 ) ,x1 = rnorm(60),x2= rnorm(60),x3=rnorm(60) ) k = kmeans( data %>% select(-group),centers = 2, iter.max = 10, nstart = 1) data$cluster = k$cluster ggplot(data ,aes(x=x1 ,y = x2,color = factor(cluster) ,shape = group ))+geom_point()
Alternatively, you can aggregated the 20 points for each group to avoid confronting the problem above. In this case each group has gets average x1 ,x2, and x3 value and those values are passed to kmeans. This works but it loses a lot of the information of the individual level data.
# clustering at the aggregated level data = data.frame(group = rep(c("A","B","C"),each =20 ) ,x1 = rnorm(60),x2= rnorm(60),x3=rnorm(60) ) data_aggregated = data %>% group_by(group) %>% summarise( avg_x1= mean(x1), avg_x2= mean(x2) , avg_x3= mean(x3) ) k = kmeans( data_aggregated %>% select(-group),centers = 2, iter.max = 10, nstart = 1) data_aggregated$cluster = k$cluster ggplot(data_aggregated ,aes(x=avg_x1 ,y = avg_x2,color = factor(cluster) ,shape = group ))+geom_point()
Any idea how to handle the first case or how the decision to use row level data or aggregated is handled?