1
$\begingroup$

I have a test dataset of 11m records. The dataset contains a global customer id and spend figure.

I need to group customers into the following categories:

  • 0 Low
  • 1 Low/Med
  • 2 Med
  • 3 Med/High
  • 4 High

I tried K-Means to group. See results below.

As you can see 10m records or so are in the low group as 80% of the db has low to negative spend.

If I want to further segment that low group, should I just increase the number of clusters? Or is there a better algorithm given the distribution of the data?

Thanks

count mean std min 25% 50% 75% Max
Cluster
0 10498822.0 21.147982 30.447597 -22885.364 6.78600 11.4520 26.30600 160.854
1 714573.0 300.654938 115.836596 160.855 207.94600 269.0280 366.02400 651.081
2 57318.0 1002.263623 400.515911 651.084 723.53375 841.8320 1118.37575 2370.803
3 14415.0 3739.988910 924.921881 2371.056 2993.61250 3599.1800 4319.69000 6162.907
4 3010.0 8584.038995 2476.616904 6163.451 6905.48800 7861.5815 9318.03100 22884.357
Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ You could well also pre-specify intervals of values for your classes in a non-data driven way, or choose the classes so that they all have the same size. Why do you think k-means is better than this? In fact the terms "low", "low/med" etc. seem to refer to the interpretation of the observed values, and k-means is not interested in this interpretation, so why do you think k-means will come up with clusters that behave well according to your pre-specified description? $\endgroup$
    – Christian Hennig
    Commented Mar 17, 2023 at 16:25
  • $\begingroup$ Thanks for the reply. The issue is that the values could change over a period of time and there are 30+ columns I need to do this for so pre-defining will be an issue. I was hoping there was a programmatic way to group in low/low-med etc. $\endgroup$
    – John Edwards
    Commented Mar 17, 2023 at 18:06
  • $\begingroup$ This depends on what you really need. k-means is defined by an objective function that may or may not deliver what you want. I suspect that out of the purely data driven clustering methods it may well be the best here, but you may want to involve side conditions that make sure it gives you something that works even better for you. But then you need to specify what's wrong with the k-means solution and what you need instead. As said already, if you want the same number of observations in each category, you can enforce this directly rather than doing k-means. $\endgroup$
    – Christian Hennig
    Commented Mar 18, 2023 at 10:49

2 Answers 2

Reset to default
1
$\begingroup$

Given your description, I would just assign cutoff points at percentiles of the distribution of total spend. With five categories, equal intervals (same number of observations in each category) would be quintiles, i.e. [0%,20%), [20%,40%),[40-60%),[60%,80%),[80%-100%].

Improve this answer
$\endgroup$
0
$\begingroup$

If you want a data driven clustering, k-means looks promising in the sense that it will produce clusters with similar within-cluster variance, which may make sense in your application. The problem of imbalanced cluster sizes may come from a very skew distribution of values, and k-means may produce something more to your liking if you transform the data first, say, by taking logs or square roots. As written in the comments, it all depends on what you really need in this situation. There is no uniquely "true" clustering that could be found reliably from data. Different clusterings are possible on the same data and background knowledge regarding the meaning of the data and aim of clustering is required to decide between them.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.