1
$\begingroup$

A colleague of mine has some monthly data that they've normalized using Z-score. So, each month, the data is normalized relative to the mean and stddev of that month and K-means is performed (K = 10). This is repeated again for the next month with that month's mean and stddev and so forth for every month. The assumption that he's made is that the cluster centers don't change that much over time and so they can be mapped to the previous month's cluster centers (based on euclidean distance).

I pointed out that while it is okay to use Z-score normalization for K-means clustering, you may not be able to compare cluster centers month-over-month as you aren't guaranteed that the normalized cluster centers actually mean the same thing. In other words, you could potentially have the same set of the values for the cluster centers from one month to the next (or some time in the next 5 years) but the distribution could have shifted its mean and stddev without affecting the Z-score.

Now, we have a nice way of mapping one set of cluster centers onto another set so that's settled. However, is there any other reasons that Z-score normalized cluster centers would prevent me from comparing/mapping cluster centers from month to month and other methods might be better (like applying a log transformation - which might squash lower values)?

$\endgroup$
0
$\begingroup$

Do not change your normalization inbetween.

Most likely, you get comparable results if you compute the normalization only once, and then apply the same, fixed normalization everywhere. Eventually, the mean will no longer be 0, but does that matter? Unless the attributes change scale independently, you should be fine (and even then you might want to capture that change...)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.