7
$\begingroup$

I ran a cluster analysis on a population of customers. I used variables like:

  • Lifetime
  • Spent amount
  • etc.

Now I'd like to use these clusters ('Little buyer', 'Regular fan of the brand A', ...) for segmentation purposes.

Problem: Future customers will appear, but I can't rerun a cluster analysis each day, so I need to assign the new customers to the current clusters.

Proposal: I can assign the future customers to the nearest centroid of the current clusters.

The new customers are assigned to clusters which have not been built for them, if the distribution of the new customers is not the same than the distribution of the current ones. Thus, the clustering is going to deteriorate.

Question: How can I monitor the evolution of the quality of the clustering / segmentation?

I was thinking about monitoring the evolution of the R-Squared (Coefficient of determination), because it's the metric I used to choose the actual clustering, but I'm not sure it's a best practice.

Improve this question
$\endgroup$
1
  • $\begingroup$ How can I add precisions in order to help you to answer the question ? $\endgroup$
    – Ophelie
    Commented Dec 4, 2013 at 16:41

1 Answer 1

Reset to default
5
$\begingroup$

Have you looked at stream clustering algorithms?

There has been some research on changing data sets, and related challenges such as concept drift.

Also, get rid of thinking in k-means terms; more modern clustering algorithms do not have spherical clusters that can be summarized with just a centroid.

Thinking of clusters as "centroids" limits your way of thinking.

Improve this answer
$\endgroup$
6
  • $\begingroup$ This seems like a promising start. Can you say more about "stream clustering", & how the OP should be thinking about how to summarize a cluster? $\endgroup$
    – gung - Reinstate Monica
    Commented Dec 4, 2013 at 22:30
  • $\begingroup$ Well, there are books on stream clustering, if I'm not mistaken... but I haven't used any of these methods, so I won't be able to give him much details; except to point out that k-means as published by MacQueen in 1967 is already streaming (but most people only know Lloyds k-means algorithm), if he really wants to stick to centroid-based models. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Dec 4, 2013 at 22:36
  • $\begingroup$ As for the second: I'd suggest to keep summarization separate from clustering. Clustering is supposed to discover structure. If you force the structure to be spherical, that is quite a strong limitation; if you just want to "summarize" your data, look for vector quantization, not structure discovery. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Dec 4, 2013 at 22:39
  • $\begingroup$ Thanks for responding so fast. This is an interesting perspective, I wonder if I should start a new thread asking about it. I think of clustering as trying to discover membership in latent categories, not structure. I wonder what the distinction is exactly, what hangs on it, etc. $\endgroup$
    – gung - Reinstate Monica
    Commented Dec 4, 2013 at 22:50
  • 1
    $\begingroup$ Well, you need to make some assumptions how how "latent categories" look like (you surely don't want them to be random; so you cannot be happy without some assumptions). If you have a good reason to assume they are spherical and that Squared Euclidean distance is the proper way of measuring similarity, then k-means is a good choice... If you assume that your latent categoires "look" different, you'll have to use a clustering algorithm that can accomodate this type of restrictions (e.g. density connected clustering such as DBSCAN). $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Dec 5, 2013 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.