5
$\begingroup$

I have several datasets with high level of skewness (many are scattered close to zero).

Applying a log transformation makes most of the data sets normally distributed.

But now how do I calculate the variance of the log-transformed datasets?

It would seem I'd have to transform back because calculating it directly on the transformed data would underestimate it. But then I'd lose the normal shape distributions, making variance calculation harder.

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ Better to say, most of the variables are more nearly normally distributed. If transformation makes sense then the SD of the logarithms also makes sense. If you want the SD on the original scale, that is always available just as it was before transformation. One is easier to think about perhaps, but less appropriate statistically and vice versa. There is literature on the lognormal that gives conversion formulas, but they don't seem much use in practice. $\endgroup$
    – Nick Cox
    Commented Oct 19, 2016 at 17:54
  • 4
    $\begingroup$ But, but, but: I can't make sense of the statement that your variables are scattered around zero, as that implies some negative values, in which case logarithmic transformation is not useful statistically. $\endgroup$
    – Nick Cox
    Commented Oct 19, 2016 at 17:56
  • $\begingroup$ They are limited to zero though, no negative values. Sorry for not making this clearer. $\endgroup$
    – DJname
    Commented Oct 19, 2016 at 18:50

1 Answer 1

Reset to default
11
$\begingroup$

Several approaches:

(i) you can estimate mean and standard deviation on both the original and the log scale as needed, in the usual fashion. However, they may not necessarily be the most efficient way on the untransformed data (nor will the two sets of estimates necessarily be very consistent with each other)

(ii) via parametric assumption -- you say on the log scale $X=\log Y$ that the distribution is approximately normal. If you assume normality of the logs, you have consistent estimates of the $\mu$ and $\sigma^2$ parameters on the log scale (indeed the usual estimates are maximum likelihood), which parameters are also the $\mu$ and $\sigma^2$ parameters of the lognormal you started with (but these are not the mean and variance of the lognormal).

You can derive the mean and variance of the lognormal in terms of those parameters easily enough, but I'll just give them:

$E(Y) = e^{\mu+\frac12 \sigma^2}$

$\text{Var}(Y) = E(Y)^2\, (e^{\sigma^2}-1)$

You obtain the standard deviation by taking the square root.

As is usually the case with MLE, these estimates are not unbiased (though they're still consistent of course). If you're especially concerned about unbiasedness, you may want some correction for small samples -- though I work with lognormal models a lot and generally just stick with MLE for the main quantities of interest.

So if you use those equations to obtain the estimates of the mean and variance of the original variable you're getting ML estimates of the parameters.

(iii) you can avoid the assumption and use Taylor expansion to get approximate moments in one direction or the other (though if you have near normality in the logs, it makes more sense to do the estimation on the log scale and transform back.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.