5
$\begingroup$

There are statistical methods (e.g. by Box-Cox or Yeo-Johnson, see references below) to automatically bring data vectors as close as possible to symmetry/normality using optimal power transformations.

Sometimes, the mentioned methods fail badly. And here I need your help. Consider for instance the distribution of $$ Y := \text{max}(100 - X, 0.0001) $$ with $X \sim \text{Exponential}(1)$. (Note that $Y := 100 - X$ except for the improbable case that $X \ge 100$ where it is put to a small positive number.) Its values are strictly positive, so Box-Cox-transformation (technically) is possible:

Example in R:

set.seed(30)
Y <- pmax(100 - rexp(1000), 0.0001)
bc <- coef(powerTransform(Y))  # 65.12778
yj <- coef(powerTransform(Y, family = "yjPower"))  # 65.78253

par(mfrow = c(3,1))
hist(Y, breaks = "FD")
hist(bcPower(Y, bc), breaks = "FD")
hist(yjPower(Y, yj), breaks = "FD")

Histograms of original variable and transformed variables

The values of $Y$ are raised to a ridiculous power of 65 and the distributions do not look very symmetric.

Of course, by studying the histogram of the untransformed variable $Y$, the problem is easily solved: We can subtract the values from about 100 and then apply Box-Cox to get a nice, symmetric result:

bc.eye <- coef(powerTransform(100 - Y))  # 0.2934301
hist(bcPower(100 - Y, bc.eye ), breaks = "FD")

Box-Cox + brain

Now finally my question: Do you know of any algorithms that chooses a linear function followed by a power transform to bring a data vector as close as possible to symmetry?

References

Box, G. E. P. and Cox, D. R. (1964) An analysis of transformations. Journal of the Royal Statistical Society, Series B. 26 211-46.

Yeo, I. and Johnson, R. (2000) A new family of power transformations to improve normality or symmetry. Biometrika, 87, 954-959.

$\endgroup$
2
  • $\begingroup$ It depends on what you want to do with the symmetrized Y. For example, do you need an estimate of scale of the symmetrized Y? $\endgroup$
    – user603
    Jun 29, 2014 at 10:30
  • $\begingroup$ Reasonable question. It is for a toy algorithm that breaks down with heavily skewed variables. You could think of something like PCA to investigate relations between possibly monotonely transformed variables. $\endgroup$
    – Michael M
    Jun 29, 2014 at 11:10

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.