1
$\begingroup$

I am doing clustering on a given data. When I plot the distributions of the individual features of this data, I found there are many columns that shows "long tail distribution".

I am wondering should I apply log transformation to these columns ?

If yes, what is the reason to do this?

I tried to conduct feature scaling only, which I would use $\frac{X−X.mean()}{X.std()}$. This should transform the values to be around 0. But I worry after the feature scaling the distribution is still so skewed that, it may be not useful to the clustering task? So I'm wondering if I should apply the log transformation before the feature scaling (standardization).

---------------------------------------------------

This is what happens when I apply log transformation: enter image description here

$\endgroup$
9
  • 1
    $\begingroup$ I do not know much about your intention for clustering, but such transformation is often useful for numerical reasons, also for results presentations, outlier detection, etc. Maybe also to find a law for the tail behavior. If the clustering is e.g. regarding distances such transformation can also help. $\endgroup$
    – user32038
    Jan 30 '19 at 12:18
  • $\begingroup$ My objective of clustering is to segment the customers into two types using the purchase/behavior data. My initial intention for the log transformation was, taking the above part of data as an example, the majority of values falls below 100, meantime there are few values are between 100 and 700, some of which may be considered as outliers. If I conduct feature scaling like normalization or min-max scaling, the majority of values between 0 and 100 will be squashed into a very small range due to the existence of the few large values. I'm not sure if that's a problem for clustering. $\endgroup$ Jan 30 '19 at 13:48
  • $\begingroup$ Usually you would standardize your variables so that you eliminate the effects of huge variances. Which is what you've done here, in a sense. So it might work, only one way to find out. $\endgroup$ Jan 30 '19 at 15:09
  • $\begingroup$ @user2974951 I think you meant feature scaling, which I used $(X-X.mean())/X.std()$. But I worry after the feature scaling the distribution is still so skewed that, it may be not useful to the clustering task? So I'm wondering if I should do the log transformation before the feature scaling (standardization). $\endgroup$ Jan 30 '19 at 15:48
  • 1
    $\begingroup$ @mkt Thank you for the comment. I'm wondering is there any article about the relations between clustering algorithms and data distributions? Thanks. $\endgroup$ Jan 31 '19 at 5:16
1
$\begingroup$

Whether you should do that depends on your task and data - we don't know either. It can be helpful, or it can harm. You need to understand your assignment to decide this.

Judging from your plot, it doesn't seem to help much.

You also seem to have 0 values, and hacked around this by using log(1+x). Why not sqrt(x)? Or pow(x, .12345)?

Instead of trying to find a transformation that makes the data distribution look pretty (which probably won't work, because you seem to have integer input, so it won't become smooth), try to better understand your problem: what do the numbers mean? How can these numbers be used to quantify similarity?

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.