I have a customer dataset, which is a survey result. I have 1595 obs. and about 200 columns (200 because most of the cases the questions were multiple choice and we had to split it into columns). Majority of variables are categorical or binary. I do not have continuous variables at all. My task is to do customer segmentation, clustering. There is no initial assumptions although as I have also the questionnaire so can logically separate the important questions.

I face several issues regarding the modeling

  1. I need to validate the choice of variables i use
  2. I am trying to find associations, pairwise associations and trends, as I do not have initial assumtions who can be my segments
  3. Clustering models are not working good for categorical variables and the ones I tried for example kmods, ignore the associations, correlations and return me not clear picture.

Can you please suggest how to approach, or from where to start. I am new in data analytics and I need some hints to go on with the analysis and I will be grateful to have some guidance at least high level what can be done.


migrated from stackoverflow.com Mar 29 '18 at 10:07

This question came from our site for professional and enthusiast programmers.


I would use a mixture model to solve this.

Each column can be represented as a realization of either a Bernoulli distribution or a categorical distribution. Assuming independence of the attributes, the observation likelihood is the product of those likelihoods and the data likelihood is the product of each of the observation likelihoods.

Now you can assume k such groupings of those Bernoulli/Categorical mixtures and find the parameters by expectation-maximization.

This may not be quick, though, and depending on your level of statistical expertise could take a while to code. I don't know of an out-of-the box package that implements something like that. I have fit mixtures of Bernoullis to cluster binary data, but I haven't done it with categorical mixed in.

It works really well and has very interpretable results unlike some other methods of binary clustering I've tried. And I find it's actually pretty robust to the independence of variables assumption. I honestly don't know why they're not more popular.


Off the top of my head:

maybe you have too many attributes and need to get rid of some to make the problem simpler and speed up the calculations. These are just ideas, not even best practices

  • determine a subset of attributes that will definitely be part of your final model (if you have a specific idea yet), keep these

  • consider removing those attributes where you have more NA values than non-NA values (maybe this holds for last questions in your questionnaire/survey?)

  • determine useless attributes (having always the same value)

  • determine highly correlated attributes, consider removing those with higher NA counts

  • decide for the remaining attributes whether the missing at random (MAR) assumption is plausible

do more stuff (I can't really tell what to do)

  • $\begingroup$ Thanks for the reply! I do have some assuptions regarding the attributes to keep in my final model, so i work on about 50 variables. I do not have NAs there, so this is my luck. My question is - how to find correlation between categorical variables (unordered categorical vs unordered categorical, and unordered categorical vs ordered categorical) the one way I read about is to find just associations with chi-square test - although I cant code it to get for all possible pairs of those 50 variables and can I take it as a base for my assumptions regarding the relation of one variable over another? $\endgroup$ – Sara Mar 29 '18 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.