I am using k-means clustering for a problem of market segmentation. My variables are gender, age and other categorical data. I first standardise age and then make all the rest dummy. The other categorical variables take approximately tens of different values, and the dataset is quite small (approx ten thousand). Then I settle on the number of clusters using the elbow method, turns out 7 should suffice. My problem is that after clustering, the majority of the resulting clusters are either composed only of male, or only of female customers; I would expect in real life this to happen only in rare circumstances. Is this behaviour expected?
$\begingroup$
$\endgroup$
5
-
$\begingroup$ You didn't make "age" a dummy variable, did you? Also how many categories are in the categorical variables, and how many categorical variables are there? How many observations? What is your k range? $\endgroup$– shadowtalkerCommented Mar 4, 2017 at 16:19
-
$\begingroup$ The reason I'm asking all those questions is that one possible cause for this behavior could be that the apparent "distances" along the gender dimension of your data are larger than a long any other dimension and are therefore dominant. $\endgroup$– shadowtalkerCommented Mar 4, 2017 at 16:30
-
$\begingroup$ @ssdecontrol, answers now included in the edit. Genders are 0 and 1, just as the other categorical variables, after I make them all dummy. $\endgroup$– famargarCommented Mar 4, 2017 at 16:30
-
$\begingroup$ You should not be using k-means if you have categorical variables (cf. here), so that's your first problem. Once you've discovered a meaningful distance function & a reasonable clustering approach, see if this problem disappears. If it doesn't, there are various troubleshooting strategies, but it might be that sex separates your data into clusters very well. You could cluster w/o sex, & see if you get the same thing. $\endgroup$– gung - Reinstate MonicaCommented Mar 4, 2017 at 16:37
-
$\begingroup$ I would be perfectly fine with a mean in gender of 0.25, i.e. 25% women. But I cannot obtain it. I did not reclusters without sex, but checked that splitting sexex does not biases too strongly the distribution of the other categorical variables. I am still at a loss. If no better ideas, I will move to hierarchical and see what I get. $\endgroup$– famargarCommented Mar 4, 2017 at 18:00
Add a comment
|