1
$\begingroup$

I have a dataset that is highly skewed. See image below: Histogram of Hydrogen Gas Untransformed

When I transform the data I get the following histogram that makes it look normal:

Log transformed Hydrogen data

This data however is not normal. I get a p-value of <2.2e-16 for the Shapiro-Wilk normality test.

Using the package distrplus in R shows that the transformed data is most likely a Gamma or a Log Normal distribution. See graphic below:

enter image description here

This data set contains over 5000 data points, is there a better way to normalise this? Can it be normalised? Or because it's so large can I assume its approximately normal due to the central limit theorem?

$\endgroup$
  • 7
    $\begingroup$ Why do you want it to be normal? $\endgroup$ – Glen_b Sep 19 '14 at 1:35
11
$\begingroup$

I have a dataset that is highly skewed. See image below: Histogram of Hydrogen Gas Untransformed

Beware drawing histograms with very few bins.

First, they're not very good at showing details of the shape, such as small modes.

Second, you can sometimes get quite misleading impressions. (It shouldn't be likely to happen with this large a data set, though)

I'd suggest:

(i) if you're going to do a histogram with so many data points, you want perhaps 4-5 times as many bins as you have; you might consider several displays at somewhat different bin-widths.

(ii) consider a kernel density estimate on the log-scale

When I transform the data I get the following histogram that makes it look normal:

Doesn't look normal to me. It looks right skew. But you need more bins.

Using the package distrplus in R shows that the transformed data is most likely a Gamma or a Log Normal distribution.

I bet you it isn't either of those. That's not to say it would be bad to use a gamma or lognormal model (such a model might be useful) - only that you'd be wrong to think your model was actually correct.

Why in your third plot is the x-axis (and the binning) different to your second plot?

This data set contains over 5000 data points, is there a better way to normalise this?

To what end?

Or because it's so large can I assume its approximately normal due to the central limit theorem?

The central limit theorem is about standardized averages as $n\to\infty$, not about the raw data. Making the sample size large makes the ECDF approach the CDF, it doesn't change the CDF at all, which will be non-normal all the way.

The most important question is What are you trying to achieve?

$\endgroup$
1
$\begingroup$

Apologies for being brutally honest, but the answer posted by the @zkurtz is entirely wrong. The way @zkurtz did it NEVER rejects the normality hypothesis since it is not testing the normality of the transformed data, but the normality of the normal distribution.

@zkurtz' procedure is like the following puzzle:

  1. Think of a number between 2 and 10.
  2. Calculate the sum of the two digits
  3. Divide the result by 3
  4. THE FINAL RESULT IS 3!!!! WOW!!!

Of course, because you are manipulating the input in such a way that you are destroying all of its information.

What you had to do in the case of x=rlnorm(5000) is to take shapiro.test(log(x)) with the usual warning for large samples.

The CLT applies to averages, not to individual observations! This is a typical puzzle for undergraduate students.

The logged-data actually looks right-skewed, which could be modeled using a skew-normal, for example. This is equivalent to fitting a log-skew-normal to the original data. See the following illustrative R code.

library(sn)

set.seed(0)
x = exp(rsn(5000,10,1,5))

hist(x,50)

hist(log(x),50)

# Data vs. Theoretical model

hist(log(x),50,probability=T)
tempf = function(x) dsn(x,10,1,5)
curve(tempf,8,15,col="red",add=T)
$\endgroup$
  • $\begingroup$ The answer this answer discusses was deleted. $\endgroup$ – Glen_b Jun 21 '16 at 8:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.