How to tell whether segments from K Means clustering result are "successful" and will impact business metrics?

Question

Background

I'm a data analyst. The Business unit I'm assigned for needs to segment users based on power vs non-power users so they can target each segment with proper treatments.

Goal

• Segment users (power vs non-power) based on their spending level (revenue generated to the company) on a small sample (20k users)
• Find the best treatment (marketing campaign, PN, etc) for each segment by experimentation

The focus of this thread is on the 1st goal (segmenting users), but it will be good to keep in mind the final goal of having these segments.

What I've done so far, in sequence

• Pick 5 features for segmentations. These are the most important business metrics, i.e. transaction amount, transaction frequency, recency.

• Clean the data, perform scaling (Min-Max Scaler, 0-1 range), and remove one highly correlated feature

• Do Elbow analysis, pretty clear that best number of cluster is 3

• Perform K Means clustering and assign a cluster for each user (PCA Viz is attached)

• Check how "good" the segments are, i.e. checking the size of each segment and difference of transaction amount per segment

• Define some thresholds based on the segment result above, so these segments can be applied to the population based on these thresholds, i.e. power user = users transacted more than 100$ and did >= 3 transactions in a month

My Questions

• The output of this segmentation is fixed thresholds, is this a best practice?

• How do I know these segments will be useful to the business? i.e. they will improve revenue once we found the best treatment for them

Basically I'm not sure how to link the K Means clustering exercise to improve the business metrics.

Answer 1

How do I know these segments will be useful to the business?

You don't. What clustering does, is split the data into groups based on a similarity metric. Cluster analysis is unsupervised, so the algorithms "decide by themselves" what kind of patterns to extract from the data, unlike supervised algorithms that are "guided" by the labels. What groups would be extracted depends on the algorithm and the data, but nothing guarantees that the algorithm would come up with the same definition of "groups of similar items" as you would. For example, if you run clustering analysis on medical X-ray images, the algorithm could come up with clusters of images that are "dark", "greyish", and "light", rather than clusters of any medical relevance. Moreover, most of the clustering algorithms will do whatever is necessary to do what you asked, so if you asked for five clusters, even if there is no meaningful way to cluster the data in five groups, it will group the data into some five groups.

Second, the results of cluster analysis need to be interpreted. Cluster analysis would tell you that the observations belong to some clusters, but it does not tell you how to understand the clusters. You noticed it yourself, to describe the clusters in plain English, you derived some thresholds like "users transacted more than 100\$ and did >= 3 transactions in a month". This is not what $k$-means does, it finds the clusters by grouping together the observations around the cluster means by the similarity metric. A better description of the groups would be to describe the "average" member of the group using the cluster means. If you want to apply the clustering to external data, rather than deriving the thresholds, use the predict method of the software that you used, which would classify the data using the similarity metrics. But going back to interpretation clusters, if you know what the cluster means, it doesn't tell you if and why those groups are distinct and if the grouping has any business relevance.

Finally, even if you have a way of meaningfully clustering the data, that is not the same as using this knowledge for making business decisions. If you group the data into business-meaningful groups, but then serve the groups with very poor marketing campaigns, you would not see any increased revenue. The clusters themselves do not tell you how to turn them into profit. If the business profits from your results, sometimes may have very little to do with the results themselves.

What I'm trying to say here is that the only thing that cluster analysis does, is groups the data into groups. It is your job to make sure that you use relevant data for the cluster analysis (so you don't accidentally cluster people on hair color or some other arbitrary criteria), do a detailed analysis to understand and validate the results (how do the clusters differ, are the patterns meaningful, what do domain experts say about the results, do the results lead to any actionable insights), and to translate the results to either insights or automated processes. So you don't know if the results would improve the revenue unless you do an experiment and try using them.