3
$\begingroup$

Is there any difference between the log transformation and standardization of data before subjecting the data to a machine learning algorithm (say k-means clustering)?

It looks like a common approach in preprocessing for clustering algorithms is to first un-skew the data through the log transformation and then perform standardization. My question is, don't both of these methods achieve the same effect when it comes to un-skewing the data? That is, both seem to transform the data into a normal distribution.

I do understand that standardization forces zero mean and unit variance but is it really required to perform both these methods on a single dataset?

So where do these two preprocessing techniques differ?

$\endgroup$

2 Answers 2

9
$\begingroup$

These two methods don't transform the data into normal distribution. And, they're very different.

  • Standardization is just making the feature zero-mean and unit variance. e.g. if the feature is uniformly distributed, it'll again be uniformly distributed. It's just a linear transform, and it doesn't decrease the skew (i.e. skewness, which is already the third standardized moment).
  • Log-transform decreases skew in some distributions, especially with large outliers. But, it may not be useful as well if the original distributed is not skewed. Also, log transform may not be applied to some cases (negative values), but standardization is always applicable (except $\sigma=0$).

The aim of stacking them together might be standardisation of all the features following the feature generation process.

$\endgroup$
3
  • $\begingroup$ Thanks ! But if log-transform decreases skew , doesn't it mean it is implicitly transforming the data into normal distribution (which has no skew ) ? $\endgroup$
    – Bharathi
    Aug 16, 2020 at 9:42
  • 2
    $\begingroup$ No, you're considering only the third moment but there are others. Transforming into another distribution means making all of them closer. For a counter-ex, think about a feature which is already normal. If you log-transform it won't be normal. $\endgroup$
    – gunes
    Aug 16, 2020 at 10:37
  • 3
    $\begingroup$ Log transformation can also increase (absolute) skew and drive a distribution further from normality. Consider the integers 1 to 100 as a start. $\endgroup$
    – Nick Cox
    Aug 16, 2020 at 11:02
8
$\begingroup$
  • Standardization does not change the skew of the distribution. What it does is transforming the values so it has $\mu=0$ and $\sigma^2=1$. The actual distribution shape remains unchanged.

  • Log-transformation, on the other hand, changes the skew of the distribution, and is useful when you deal with values that have right-tailed distribution.


Consider the following example.

set.seed(0)
X <- rlnorm(1000)
hist(X)

enter image description here

You can see that X has skewed distribution (long right tail).

Now we do two transformations as follows:

Z <- (X-mean(X))/sd(X)
L <- log(X)
par(mfrow = c(1,2))
hist(Z)
hist(L)

enter image description here

As you can see, standardization of X (which is Z in this example) has similar shape of distribution with X. Log-transformation of X (which is L in this example) is now transformed in a way that eliminates its long right tail.

Note that X in the previous example has lognormal distribution X <- rlnorm(1000), that's why the log-transformed variable looks normally distributed. If X is, say, exponentially distributed, the resulting log-transformed variable may not be normal, as shown below.

set.seed(0)
X <- rexp(1000)
par(mfrow = c(1,1))
hist(X)

enter image description here

Z <- (X-mean(X))/sd(X)
L <- log(X)
par(mfrow = c(1,2))
hist(Z)
hist(L)

enter image description here

You can see that the distribution of log-transformed X (denoted by L) now does not have right tail, but still not looking like normal distribution because it's still skewed. Log-transformation does not always make the distribution normal.


$\endgroup$
6
  • $\begingroup$ Great explanation ! Thanks , but I don't understand how log transform does not transform the data to normal distribution . You say it unskews the data but doesn't that mean kind of centering the data towards normal distribution ? Please help me $\endgroup$
    – Bharathi
    Aug 16, 2020 at 9:44
  • $\begingroup$ Could you also justify why log-transforming an exponential distribution may not be normal ? Thank you $\endgroup$
    – Bharathi
    Aug 16, 2020 at 9:53
  • $\begingroup$ @Bharathi it will not always bring the distribution to normal. It removes right-tail skew, but not necessarily make the distribution normal. I'm improving the answer to reflect this. $\endgroup$ Aug 16, 2020 at 14:10
  • $\begingroup$ Okay great ! So you mean , log-transformation of left-skewed dist can sometime also make it right-skewed, right ? And not necessarily normal though it reduces skew $\endgroup$
    – Bharathi
    Aug 16, 2020 at 15:34
  • 2
    $\begingroup$ I can't think of an example in which a left-skewed distribution is made right-skewed by logarithms. Such distributions you'd expect to become more left-skewed usually, as in relative terms logarithms stretch low values and squeeze high values. There is an interesting case of a distribution with just two distinct positive values. Its skewness is unchanged under logarithms even if one value is an extreme outlier. That is easy to see graphically as the histogram must remain as two spikes and so the shape doesn't change. In general, don't guess wildly: try it out with (simple) examples. $\endgroup$
    – Nick Cox
    Aug 17, 2020 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.