customer segmentation with categorical variables

Question

I have a customer dataset, which is a survey result. I have 1595 obs. and about 200 columns (200 because most of the cases the questions were multiple choice and we had to split it into columns). Majority of variables are categorical or binary. I do not have continuous variables at all. My task is to do customer segmentation, clustering. There is no initial assumptions although as I have also the questionnaire so can logically separate the important questions.

I face several issues regarding the modeling

1. I need to validate the choice of variables i use
2. I am trying to find associations, pairwise associations and trends, as I do not have initial assumtions who can be my segments
3. Clustering models are not working good for categorical variables and the ones I tried for example kmods, ignore the associations, correlations and return me not clear picture.

Can you please suggest how to approach, or from where to start. I am new in data analytics and I need some hints to go on with the analysis and I will be grateful to have some guidance at least high level what can be done.

Answer 1

I would use a mixture model to solve this.

Each column can be represented as a realization of either a Bernoulli distribution or a categorical distribution. Assuming independence of the attributes, the observation likelihood is the product of those likelihoods and the data likelihood is the product of each of the observation likelihoods.

Now you can assume k such groupings of those Bernoulli/Categorical mixtures and find the parameters by expectation-maximization.

This may not be quick, though, and depending on your level of statistical expertise could take a while to code. I don't know of an out-of-the box package that implements something like that. I have fit mixtures of Bernoullis to cluster binary data, but I haven't done it with categorical mixed in.

It works really well and has very interpretable results unlike some other methods of binary clustering I've tried. And I find it's actually pretty robust to the independence of variables assumption. I honestly don't know why they're not more popular.

Answer 2

Off the top of my head:

maybe you have too many attributes and need to get rid of some to make the problem simpler and speed up the calculations. These are just ideas, not even best practices

• determine a subset of attributes that will definitely be part of your final model (if you have a specific idea yet), keep these

• consider removing those attributes where you have more NA values than non-NA values (maybe this holds for last questions in your questionnaire/survey?)

• determine useless attributes (having always the same value)

• determine highly correlated attributes, consider removing those with higher NA counts

• decide for the remaining attributes whether the missing at random (MAR) assumption is plausible

do more stuff (I can't really tell what to do)

Answer 3

Often times (actually, most of the time) data can contain just as many categorical variables as continuous ones. Regardless, the task is to compute customer segmentation analysis. In parallel, you need to detect associations and correlations with categorical variables - a good lead-in to customer segmentation, anyway.

So some approaches:

1. Evaluate k-modes again, but first use a dimensionality reduction technique like principal component analysis to extract a few set of variables that are explaining the maximum variance in the dataset. Hard to "visualize" k-modes outputs if it's too cluttered. Take it one variable at a time.
2. Use chi-square tests to evaluate associations between 2 categorical variables. Beware that in high-count comparisons, very similar groups can still yield statistically significant differences (though they aren't), so good to apply a Cramer's V correction to it. But once associations evaluated here, can layer on chi-square automated interaction detector (CHAID). It's a decision tree classifier that creates groupers. And from this you can evaluate segments.
3. Logistic regressions also work well, and most importantly, is interpretable. Look at significant odds-ratios. Remember to understand which of the variables is serving as the baseline (e.g., when looking at gender, males are 1.5x more likely than females to get/have/etc "y" - whatever your outcome "y" is).
4. Evaluate cosign similarity

Last, though very useful in other cases, here I would not recommend one-hot encoding to generate numerical values just to get them working in a k-means algorithm. It's forcing a category to be a number, when there's no inherent Euclidean distance between "cat" and "dog", or "Boston" and "San Francisco." They're different, yes. But how different numerically? Impossible to know, except in the sense of frequency distributions with respect to another variable. One-hot encoding will simply enforce a category to be 0 or 1, which are relatively extreme values and will make models apply more weight to them than continuous variables when both are introduced simultaneously into your model.