I want to perform cluster analysis based on three values for each observation. All these values are extremely skewed to the right. they are the number of friends, followers, and statuses of a person in twitter and vary between 0 to 1000000. I know that I should first normalize my data, but I don't know what kind of normalization leads to better results in this case.

Nip asks a similar question, but I guess the difference between my question and his is that my data is not normally distributed, and I don't know if standardization can be used at all.

EDIT 1: I log-transformmed the data, but I still have zero inflation. Anderson-Darling test p value is less than 0.05, and hence we reject the null hypotheses that the data are from a normal distribution.

Thank you

  • $\begingroup$ Don't worry about non normality. When your data is just normal, that's the case in which you don't want to perform clustering $\endgroup$ – carlo Oct 26 '19 at 10:36

log transform, then standardize.

I would think that if someone has 100/200 contacts, they "play in the same league" with people who have 500/1000. Instead, people who have 100'000 contacts, "plays in the same league" with those who have 500'000, and probably are trying to get to 1'000'000. Logarithmic transformation will make your data reflect this.

  • $\begingroup$ My data looks much nicer after log transform, but I still have a "ceiling effect':a big chunk of data, even after transformation, remains at the "zero" bin. Can I standardize my data despite this effect now? $\endgroup$ – Pie-ton Oct 25 '19 at 23:08
  • $\begingroup$ yes, no problem. clustering will probably keep most of them together, and that's fine. that's nothing left to do now, apart from standardization. $\endgroup$ – carlo Oct 25 '19 at 23:55
  • $\begingroup$ The large number of zeros is not a ceiling effect. A ceiling effect would be like scores on an easy test where a lot of people score at or near 100. You might have zero-inflated data, a common issue with counts; transformation can never deal with a big spike at 0 because all the zeros will stay together. It's not clear why you need to "standardize" anything $\endgroup$ – Glen_b -Reinstate Monica Oct 26 '19 at 4:44
  • $\begingroup$ @Glen_b: I stand corrected. I meant "zero inflation." I do not necessarily want to "standardize" the data. I want to cluster my data, and I know I should do some form of "normalization" before clustering, but I don't know if I should standardize, normalize, or do anything else. So I guess my main question is : Before clustering, should I do any kind of transformation, normalization etc. for a data set with the characteristics I explained above (zero inflation, Poisson-like distribution, with 1,100,000 observations? If yes, what is the proper way to do that? $\endgroup$ – Pie-ton Oct 26 '19 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.