12
$\begingroup$

Suppose I have a leptokurtic variable that I would like to transform to normality. What transformations can accomplish this task? I am well aware that transforming data may not always be desirable, but as an academic pursuit, suppose I want to "hammer" the data into normality. Additionally, as you can tell from the plot, all values are strictly positive.

I have tried a variety of transformations (pretty much anything I have seen used before, including $\frac 1 X,\sqrt X,\text{asinh}(X)$, etc.), but none of them work particularly well. Are there well-known transformations for making leptokurtic distributions more normal?

See the example Normal Q-Q plot below:

enter image description here

$\endgroup$
  • 5
    $\begingroup$ Are you familiar with the probability integral transform? It has been invoked in a few threads on this site, if you would like to see it in action. $\endgroup$ – whuber Feb 6 '14 at 22:57
  • 8
    $\begingroup$ You need something that works symmetrically on (variable $-$ "middle") while also respecting sign. Nothing that you tried comes close if you don't have a "middle". Use median for "middle" and try cube root of the deviations, remembering to implement cube root as sign(.) * abs(.)^(1/3). No guarantees and very ad hoc, but it should push in the right direction. $\endgroup$ – Nick Cox Feb 6 '14 at 23:05
  • 1
    $\begingroup$ Uh, what makes you call that platykurtic? Unless I missed something, that looks like it's got higher kurtosis than the normal. $\endgroup$ – Glen_b Feb 6 '14 at 23:10
  • 3
    $\begingroup$ @Glen_b I think is right: it's leptokurtic. But both these terms are pretty silly, except in so far as they allow reference to the original cartoon by Student in Biometrika. The criterion is kurtosis; values are high or low or (even better) quantified. $\endgroup$ – Nick Cox Feb 6 '14 at 23:26
  • 3
    $\begingroup$ Why is leptokurtic described as 'thin tailed'? While there's no necessary relationship between thickness of tail and kurtosis, the general tendency is for heavy tails to be associated with kurtosis (e.g. compare $t_6$ with normal, for standardized densities) $\endgroup$ – Glen_b Feb 7 '14 at 2:09
11
$\begingroup$

I use heavy tail Lambert W x F distributions to describe and transform leptokurtic data. See (my) following posts for more details and references:

Here is a reproducible example using the LambertW R package.

library(LambertW)
set.seed(1)
theta.tmp <- list(beta = c(2000, 400), delta = 0.2)
yy <- rLambertW(n = 100, distname = "normal", 
                theta = theta.tmp)

test_norm(yy)

test normality of original data

## $seed
## [1] 267509
## 
## $shapiro.wilk
## 
##  Shapiro-Wilk normality test
## 
## data:  data.test
## W = 1, p-value = 0.008
## 
## 
## $shapiro.francia
## 
## 	Shapiro-Francia normality test
## 
## data:  data.test
## W = 1, p-value = 0.003
## 
## 
## $anderson.darling
## 
##  Anderson-Darling normality test
## 
## data:  data
## A = 1, p-value = 0.01

The qqplot of yy is very close to your qqplot in the original post and the data is indeed slightly leptokurtic with a kurtosis of 5. Hence your data can be well described by a Lambert W $\times$ Gaussian distribution with input $X \sim N (2000, 400)$ and a tail parameter of $\delta = 0.2$ (which implies that only moments up to order $\leq 5$ exist).

Now back to your question: how to make this leptokurtic data normal again? Well, we can estimate the parameters of the distribution using MLE (or for methods of moments use IGMM()),

mod.Lh <- MLE_LambertW(yy, distname = "normal", type = "h")
summary(mod.Lh)

## Call: MLE_LambertW(y = yy, distname = "normal", type = "h")
## Estimation method: MLE
## Input distribution: normal
## 
##  Parameter estimates:
##        Estimate  Std. Error  t value Pr(>|t|)    
## mu     2.05e+03    4.03e+01    50.88   <2e-16 ***
## sigma  3.64e+02    4.36e+01     8.37   <2e-16 ***
## delta  1.64e-01    7.84e-02     2.09    0.037 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## -------------------------------------------------------------- 
## 
## Given these input parameter estimates the moments of the output random variable are 
##   (assuming Gaussian input): 
##  mu_y = 2052; sigma_y = 491; skewness = 0; kurtosis = 13.

and then use the bijective inverse transformation (based on W_delta()) to backtransform the data to the input $X$, which -- by design -- should be very close to a normal.

# get_input() handles does the right transformations automatically based on
# estimates in mod.Lh
xx <- get_input(mod.Lh)
test_norm(xx)

test normality of Gaussianized data

## $seed
## [1] 218646
## 
## $shapiro.wilk
## 
##  Shapiro-Wilk normality test
## 
## data:  data.test
## W = 1, p-value = 1
## 
## 
## $shapiro.francia
## 
## 	Shapiro-Francia normality test
## 
## data:  data.test
## W = 1, p-value = 1
## 
## 
## $anderson.darling
## 
##  Anderson-Darling normality test
## 
## data:  data
## A = 0.1, p-value = 1

Voila!

$\endgroup$
9
$\begingroup$

Credit for this answer goes to @NickCox's suggestion in the comments section of the original question. He suggested I subtract the median of the data and apply the transformation to the deviations. For instance, $\text{sign(.)}\cdot\text{abs(.)}^{\frac 1 3}$, with $Y-\text{median}(Y)$ as the argument.

Although the cube root transformation didn't work out well, it turns out the square root and the more obscure three-quarters root work well.

Here was the original kernel density plot corresponding to the Q-Q plot of the leptokurtic variable in the original question:

enter image description here


After applying the square root transformation to the deviations, the Q-Q plot looks like this:

enter image description here

Better, but it can be closer.


Hammering some more, applying the three-quarters root transformation to the deviations gives:

enter image description here


And the final kernel density of this transformed variable looks like this:

enter image description here

Looks close to me.

$\endgroup$
8
$\begingroup$

In many cases, there may simply be no simple-form monotonic transformation that will produce a close-to-normal result.

For example, imagine that we have a distribution which is a finite mixture of lognormal distributions of various parameters. A log transform would transform any of the components of the mixture to normality, but the mixture of normals in the transformed data leaves you with something that's not normal.

Or there may be relatively nice transform, but not of one of the forms you'd think to try -- if you don't know the distribution of the data, you may not find it. For example, if the data were gamma-distributed, you won't even find the exact transform to normality (which certainly exists) unless I tell you exactly what the distribution is (though you might stumble upon the cube-root transformation that in this case would make it pretty close to normal as long as the shape parameter isn't too small).

There are myriad ways in which the data can look reasonably amenable to being transformed but which doesn't look great on any of a list of obvious transformations.

If you can give us access to the data, it may well be that we can either spot a transformation that does okay -- or that we can show you why you won't find one.

Just from the visual impression there, it looks rather like a mixture of two normals with different scales. There's only a slight hint of asymmetry, which you could easily observe by chance. Here's an example of a sample from a mixture of two normals with common mean - as you see it looks quite a bit like your plot (but other samples may look heavier or lighter tailed - at this sample size there's a lot of variation in the order statistics outside 1 sd either side of the mean).

enter image description here

In fact here are yours and mine superimposed:

$\quad\quad\quad $enter image description here

$\endgroup$
  • $\begingroup$ It certainly does look like a mixture of two normal distributions of different scale, excellent observation. $\endgroup$ – Underminer Feb 7 '14 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.