0
$\begingroup$

So I was playing with some data today, and I plotted a histogram of it. I obtained the following distribution:

enter image description here

Incredibly skewed! To fix this skewness, it makes sense to take the natural logarithm of the distribution:

enter image description here

Okay - now the distribution doesn't look so normal. Taking the log didn't remove any skew. When does takibng the logarithm - not remove any skew?

Improve this question
$\endgroup$
8
  • 3
    $\begingroup$ ??? It is evident from the graphs that it is less skew than it was and it is closer to normal, so why say otherwise? Measures of skewness can be a little tricky, but I guarantee you that any common measure of skewness (moment-based; (mean - median) / SD, etc.) will show that the second distribution is closer to symmetry. If anyone promises you in a textbook, paper or presentation that logarithmic transformation will always remove skewness, that is wrong. $\endgroup$
    – Nick Cox
    Mar 23, 2016 at 15:26
  • 3
    $\begingroup$ You said, and the question text still says, "didn't remove any skew", which is wrong. I agree that it's not normal and didn't claim otherwise. By definition (quite literally) the only distribution which logarithmic transformation will render perfectly normal is the lognormal and it is just of many, many right-skewed distributions. To avoid mixing related, but not identical, questions, note that normal distributions are all symmetric but not all symmetric distributions are normal, so that removing skew and rendering normal are not one and the same. $\endgroup$
    – Nick Cox
    Mar 23, 2016 at 15:45
  • 4
    $\begingroup$ Zero, your questions exhibit an unfortunate tendency to make strong, false declarations. This makes them contentious and difficult to address: should people respond to the false assumptions, or should they guess what you're really trying to get at? In this case, for instance, I would suppose you are trying to find a transformation that will make this distribution more symmetric and were surprised that the log did not succeed. Why not say those things directly? You will get better answers more quickly (and we will not need to work so hard to understand what you are trying to ask). $\endgroup$
    – whuber ♦
    Mar 23, 2016 at 15:55
  • 2
    $\begingroup$ another question based on a false premise unfortunately; clearly the second one is less skew than the first. As whuber says, this makes the questions hard to answer. Describe what you want to achieve and then ask questions (e.g. "why didn't the histogram end up looking more symmetric than this?") where you make statements. You'll end up with much more informative answers. $\endgroup$
    – Glen_b
    Mar 24, 2016 at 0:44
  • 1
    $\begingroup$ See this post: stats.stackexchange.com/questions/107610/… -- my answer there shows histograms of data from three fairly similar-looking right skewed distributions, which after taking logs, you get left skew, symmetry and (reduced) right skew, as well as some explanation of what's going on. [NB If you're trying to do regression, none of the variables need themselves be symmetric. There may sometimes be reason to transform DVs or IVs, but that's not one of them.] $\endgroup$
    – Glen_b
    Mar 24, 2016 at 0:46

1 Answer 1

Reset to default
4
$\begingroup$

The data come from here and are evidently people's ages. The precise age range is from 19 to 75 years. There are 1000 values.

The same data also appear in earlier threads started by @zero:

If the mean and median underestimate the true central tendency, why use them?

Using MLE to determine parameters for QQ plot

I am always eager to see graphs first, but some numerical results are relevant to the question too. In addition to considering the logarithmic transformation, the lower limit well above 0 implies that we should also be considering transformations of the form log(age $-\ k$) for age $>k$. Indeed @zero in the second thread (see above) used a three-parameter lognormal, itself implying that such a transformation is more appropriate.

For concreteness, I tried $k =$ 18.

Here are the conventional moment-based measures as calculated in Stata. The formulas used are documented here and also discussed here. That is likely to be unimportant except that other software subtracts 3 in presenting kurtosis, and other software may otherwise use slightly different formulas. Most crucially, with the definition used, a normal or Gaussian would have kurtosis 3.

The column headed (*) is (mean $-$ median)/SD which has the convenient virtue of being bounded by $-$1 and 1. More obviously, it is 0 if and only if mean = median. Sample skewness and kurtosis are bounded by functions of sample size, but those limits don't bite here: for example, with 1000 values skewness can't exceed 31.606 (3 d.p.). 

 n = 1000       |       mean          SD    skewness    kurtosis     (*)
----------------+--------------------------------------------------------
            age |     35.542      11.353       1.023       3.611    0.224
        log age |      3.524       0.299       0.414       2.453    0.093 
 log (age - 18) |      2.641       0.706      -0.442       2.970   -0.095 
-------------------------------------------------------------------------

What wording you want to apply is a matter of taste as well as experience, but I would suggest that the raw data are moderately skewed (I wouldn't use @zero's wording "incredibly" at all), logarithmic transformation helps, and the detail of whether you subtract a constant first is important.

In terms of the original question:

  1. The logarithmic transformation is not as ineffective as implied, especially when you generalise it.

  2. It remains true that there is no guarantee that logarithmic transformation will symmetrize, let alone render normal, any distribution, even if we restrict discussion, as we should, to variables that are all positive before we transform.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.