I often hear that comparing clustering computed on different population is mathematically incorrect. However, I never found any references about it.

Imagine I am working on a dataset such as

  household_id  sex  var1 var2 var3
            1   man    x    y    z
            1 woman    x    y    z
            2   man    x    y    z
            2 woman    x    y    z
            3   man    x    y    z
            3 woman    x    y    z

where household_id is a personal identifier of an household, and var_a set of variables I'm interested in.

I have two related individuals in the household (siblings, couple, parent-children, ...).

I would like to use clustering techniques on the var_ to come up with a typology.

My issue is this. What if I want to analyse men and women separately. What I would do is to would compute separately for men and women the clustering and then compare the clusters. In cross-tabulating them for example. However, I was told this is wrong. Can you explain me why exactly ?

Moreover, what would be an alternative ? I feel that I am loosing a lot of important differences between the genders in running the clustering on the full database. Also, man and woman are related in this data, and running the analysis on the full database is simply ignoring this household level.

Let me provide an empirical example (with a R code).

Let us imagine I want to compare sequences of men and women

   idhouse   sex V1 V2 V3 V4 V5
1        1   man  a  a  b  c  b
2        1 woman  a  c  a  a  c
3        2 woman  a  c  a  a  c
4        2   man  b  c  c  c  a

I will run my sequence analysis and my clustering like :

sq_man = seqdef(df[df$sex == 'man',-c(1,2)])
sq_wom = seqdef(df[df$sex == 'woman',-c(1,2)])

sq_manHAM = seqdist(sq_man, 'HAM')
sq_womHAM = seqdist(sq_wom, 'HAM')

w1 = hclust(as.dist(sq_manHAM), 'ward')
w2 = hclust(as.dist(sq_womHAM), 'ward')

and end up with a typology for men and a typology for women

men = cutree(w1, 2)
wom = cutree(w2, 3)

Then for example comparing these with (cross-table)

table(men, wom)

   men 1 2 3
     1 2 0 0
     2 0 1 2

is not proper. Does anyone knows why ?



df = replicate(5, sample(x = c('a', 'b', 'c'), 10, replace = T))
df = as.data.frame(df)
df$idhouse = rep(1:5, 2)
df$sex = rep(c('man', 'woman'), 5)
df = arrange(df, idhouse)
df = select(df, idhouse, sex, everything())
  • $\begingroup$ Can you give a reference for the claim? $\endgroup$ Dec 7, 2016 at 11:39
  • $\begingroup$ and then compare the clusters It is unclear what you mean by comparing clustering results of different data. What is your aim and your proposed plan in that? A by note: one should not normally di Ward clustering on matrix of Gower distance. Gower coefficient isn't euclidean distance, even not metric. $\endgroup$
    – ttnphns
    Dec 7, 2016 at 12:16


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.