1
$\begingroup$

I have the following distribution:

enter image description here

I have numerous zero values which I change to 1 so I don't have zero or negative numbers before I log transform. Of course, log(1) = 0. So I get the following results:

enter image description here

Of course, the log transform has given me more of a normal distribution except for the lower end of the data. Does the spike mean that performing the transformation is meaningless?

Improve this question
$\endgroup$
11
  • $\begingroup$ What problem are you trying to solve? $\endgroup$
    – Sycorax ♦
    Commented Jan 16, 2020 at 17:01
  • $\begingroup$ This is input data. Many models work best when the input is normally distributed. When faced with an exponential distribution I apply a log transformation. But the lower end value give me the spike in the otherwise more normally distributed data. Therefore, does it make sense to NOT transform this data? $\endgroup$
    – Windstorm1981
    Commented Jan 16, 2020 at 17:03
  • 2
    $\begingroup$ We find many people asking questions predicated on this "common practice"--but it's usually based on the wrong considerations. "Parametric" is not the same as "Gaussian;" response variables usually ought to have complex distributions that reflect their explanatory variables; most parametric assumptions pertain to error terms, not to the variables themselves; whether and how to transform a variable depends on how it will be used in an analysis; and much more. Thus, you might or might not have a problem here and your approach might make the situation better, worse, or not matter. $\endgroup$
    – whuber ♦
    Commented Jan 16, 2020 at 17:52
  • 2
    $\begingroup$ Some relevant threads include stats.stackexchange.com/questions/30728 (on taking logs of data with zero values), stats.stackexchange.com/questions/35711 (on transforming a response variable), and stats.stackexchange.com/questions/4831 (on transforming regression variables). $\endgroup$
    – whuber ♦
    Commented Jan 16, 2020 at 17:55
  • 1
    $\begingroup$ @whuber your suggestion stats.stackexchange.com/questions/4831/… is very helpful $\endgroup$
    – Windstorm1981
    Commented Jan 16, 2020 at 18:31

2 Answers 2

Reset to default
2
$\begingroup$

If the response is count valued, you should consider using an appropriate modeling strategy that implicitly log-transforms the intensity rather than the count values themselves. Consider that a Poisson process with low intensity is likely to have right skewed results and many 0s, but log-transforms of the data would lead to highly biased estimates of the mean and of the SD, regardless of how the 0 values are handled. Fitting negative binomial or quasipoisson GLM as probabilistic models via MLE is a simple, straightforward, and efficient way to summarize count data where applicable.

A log-transform alternative that has been explored for variables with 0 values is logp1, where 1 is added to every value before applying log transform, and not just to the 0 value. This will decrease the intensity of the truncation, but not satisfactorily. In other words, your approach is not justified regardless of the distribution's appearance post hoc. The resulting scatterplot is evidence of this.

If the response is continuously valued, the distribution has mixture properties and mixture modeling is one approach to estimating a density with better precision that the proposed approach.

Improve this answer
$\endgroup$
0
$\begingroup$

One possibility is to separate your data into 2 variables. One with an indicator of the value is zero or non zero. And the other being value of the variable given a non zero variable. And then apply the transformation. On a separate note you can try using power transform/box Cox transform.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Boxcox with unspecified Lambda on this series gave a value of 0.04 - which is virtually the log transformation. $\endgroup$
    – Windstorm1981
    Commented Jan 16, 2020 at 17:27
  • $\begingroup$ True. But it should be remembered that Box Cox transformations have a limitation that data should be positive which is applicable for log transformation as well. So better approach is to remove non positive data points and keep them in a separate variable which should make your data probably an exponential. $\endgroup$
    – Charles Smith
    Commented Jan 16, 2020 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.