I often hear that comparing clustering computed on different population is mathematically incorrect. However, I never found any references about it.
Imagine I am working on a dataset such as
household_id sex var1 var2 var3
1 man x y z
1 woman x y z
2 man x y z
2 woman x y z
3 man x y z
3 woman x y z
where household_id
is a personal identifier of an household, and var_
a set of variables I'm interested in.
I have two related individuals in the household (siblings, couple, parent-children, ...).
I would like to use clustering techniques on the var_
to come up with a typology.
My issue is this. What if I want to analyse men and women separately. What I would do is to would compute separately for men and women the clustering and then compare the clusters. In cross-tabulating them for example. However, I was told this is wrong. Can you explain me why exactly ?
Moreover, what would be an alternative ? I feel that I am loosing a lot of important differences between the genders in running the clustering on the full database. Also, man and woman are related in this data, and running the analysis on the full database is simply ignoring this household level.
Let me provide an empirical example (with a R
code).
Let us imagine I want to compare sequences of men and women
idhouse sex V1 V2 V3 V4 V5
1 1 man a a b c b
2 1 woman a c a a c
3 2 woman a c a a c
4 2 man b c c c a
I will run my sequence analysis and my clustering like :
sq_man = seqdef(df[df$sex == 'man',-c(1,2)])
sq_wom = seqdef(df[df$sex == 'woman',-c(1,2)])
sq_manHAM = seqdist(sq_man, 'HAM')
sq_womHAM = seqdist(sq_wom, 'HAM')
w1 = hclust(as.dist(sq_manHAM), 'ward')
w2 = hclust(as.dist(sq_womHAM), 'ward')
and end up with a typology for men and a typology for women
men = cutree(w1, 2)
wom = cutree(w2, 3)
Then for example comparing these with (cross-table)
table(men, wom)
wom
men 1 2 3
1 2 0 0
2 0 1 2
is not proper. Does anyone knows why ?
data
library(dplyr)
library(TraMineR)
df = replicate(5, sample(x = c('a', 'b', 'c'), 10, replace = T))
df = as.data.frame(df)
df$idhouse = rep(1:5, 2)
df$sex = rep(c('man', 'woman'), 5)
df = arrange(df, idhouse)
df = select(df, idhouse, sex, everything())
and then compare the clusters
It is unclear what you mean by comparing clustering results of different data. What is your aim and your proposed plan in that? A by note: one should not normally di Ward clustering on matrix of Gower distance. Gower coefficient isn't euclidean distance, even not metric. $\endgroup$