Assume that I have a variable whose distribution is skewed positively to a very high degree, such that taking the log will not be sufficient in order to bring it within the range of skewness for a normal distribution. What are my options at this point? What can I do to transform the variable to a normal distribution?
$\begingroup$
$\endgroup$
16
-
2$\begingroup$ Just to make sure, "skewed negative" means the long tail pointing to left or right? If it's really skewed negatively (long tail left), log-transformation wouldn't work very well. $\endgroup$– Penguin_KnightFeb 28, 2014 at 1:28
-
7$\begingroup$ Reciprocal transformation is stronger than logarithmic and often preserves meaning, as units of measurement are just inverted. For example, the reciprocal of the time to do something is a kind of speed, and vice versa. The reciprocal of miles per gallon or km per litre makes sense. Reciprocals invert the order and can be negated if that is preferable. They are naturally part of the Box-Cox scheme with that extra detail. All values should be positive for this to work well. (In principle, it would work with all values negative, but I've yet to see an example in practice.) $\endgroup$– Nick CoxFeb 28, 2014 at 7:39
-
3$\begingroup$ @Aksakal $\ln(\ln())$ I can't see as a good idea. The result is statistically meaningful only for values $>1$. If values are counts, it is artificial that a transform be undefined for 0s or 1s, regardless of whether those values occur in the data. If values are measurements the restriction means that the validity of a transformation depends on the choice of units of measurement, which is absurd, as if $\ln(\ln(0.7))$ can't be done because I use cm, but $\ln(\ln(7))$ can be done because I use mm. (That logarithms yield complex results for negative arguments I don't think helps statistically.) $\endgroup$– Nick CoxFeb 28, 2014 at 10:13
-
3$\begingroup$ @Aksakal Too strong to say "log transform is not a tool to cure the skewness": if skewness is the only issue, logs often work very well. If your point is that skewness of marginal distributions need not be a major problem, I tend to agree. $\endgroup$– Nick CoxFeb 28, 2014 at 10:22
-
4$\begingroup$ I naturally agree, but if I used squares or logarithms, I wouldn't feel obliged to offer references, and similarly here. But the usefulness of reciprocals, particularly times and speeds, was stressed by (e.g.) Tukey, J.W. 1977. Exploratory data analysis. Reading, MA: Addison-Wesley and in several of his papers. Miles per gallon and gallons per mile (or conversely litres per km and km per litre) are common place in discussions of car performance data. Densities and their reciprocals are fairly standard examples in geography and demography. $\endgroup$– Nick CoxOct 19, 2015 at 14:55
|
Show 11 more comments
3 Answers
$\begingroup$
$\endgroup$
Try straight Box-Cox transform as per Box, G. E. P. and Cox, D. R. (1964), "An Analysis of Transformations," Journal of the Royal Statistical Society, Series B, 26, 211--234. SAS has the description of its loglikelihood function in Normalizing Transformations, which you can use to find the optimal $\lambda$ parameter, which is described in Atkinson, A. C. (1985), Plots, Transformations, and Regression, New York: Oxford University Press.
It's very easy to implement it having the LL function, or if you have a stat package like SAS or MATLAB use their commands: it's boxcox command in MATLAB and PROC TRANSREG in SAS.
Also, in R this is in the MASS package, function boxcox().
$\begingroup$
$\endgroup$
For positive skew (tail is on the positive end of the x axis), there are the square root transformation, the log transformation, and the inverse/reciprocal transformation (in order of increasing severity). Thus, if the log transformation is not sufficient, you can use the next level of transformation. Box Cox runs all transformations automatically so you can choose the best one.
answered Feb 28, 2014 at 14:56
$\begingroup$
$\endgroup$
1
Most software suites will use Euler's number as the default log base, AKA: natural log. You can use a higher base number to rein in excessively right-skewed data. How you do it syntax-wise depends on the software you are using.
If you need to get back out of you transformed values once estimations have been done, it might be a little easier to use this method because all you have to do is perform a exponential operator on your variable with whatever your log base was.
-
9$\begingroup$ This makes no sense at all. The logarithms to two different bases differ only by a multiplicative constant and the skewness reduction by either is thus the same. Thus 1 10 100 1000 10000 is symmetrical after transforming log base 10 and it would be just as symmetrical after log base $e$ or log base 2. The only difference is a scaling factor. $\endgroup$– Nick CoxJun 9, 2016 at 21:46