2
$\begingroup$

I have a dataset of metric distances ($n=5800$) and plotted those as a histogram. My initial thought was that this distribution looks normal. But after performing a Shapiro Wilk test and plotting the distribution like below I am not that sure anymore. In the Shapiro Wilk test the p value was way to low.

My question is: Can this distribution still be considered as a Gaussian distribution? The mean of my data is -0.027 and standard deviation is 0.72.

enter image description here

Also a qq plot doesn't look like a normal distribution to me. But I am also not sure how to define this distribution.

enter image description here

When limiting my data range to -1.4 and 1.4 the qq plot looks like the following. How can this be interpreted?

enter image description here

Edit: Did a new plot with different software this time. Now the plot doesn't look normally distributed to me anymore.

enter image description here

$\endgroup$
10
  • 6
    $\begingroup$ The more important question is: Why would normality be important to you? What are you trying to find out with these data? $\endgroup$ Commented Nov 6, 2018 at 18:40
  • 2
    $\begingroup$ The probability plot is inconsistent with the density plot at the outset. $\endgroup$
    – whuber
    Commented Nov 6, 2018 at 18:51
  • 1
    $\begingroup$ The most likely reasons are that the density plot was computed from other data or else was not computed using an appropriate algorithm. $\endgroup$
    – whuber
    Commented Nov 6, 2018 at 21:03
  • 1
    $\begingroup$ Ok, I see what you mean. I did another plot, this time using the python package seaborn. Now the density plot is not looking that normally distributed anymore. Added the new plot to my question. $\endgroup$
    – conste
    Commented Nov 6, 2018 at 21:40
  • 2
    $\begingroup$ This leads me to suspect the original plot might have been a density estimate using a Gaussian kernel with overly wide bandwidth. The kernel dominated the data, making them look almost Normal. It's a nice example of why probability plots are superior to histograms and density plots for understanding the details of data distributions. $\endgroup$
    – whuber
    Commented Nov 6, 2018 at 21:47

3 Answers 3

2
$\begingroup$

Your first plot is a good example that histograms or density estimates (if using too much smoothing) is a poor way of identifying distributions. I first made a histogram with many small bins, to my eyes that do not look normal (not shown). The details can be seen much clearer if showing the histogram on a log scale:

Histogram on a log scale

with the best-fitting normal density superposed. R Code is here:

    myhist <- hist(ydata, prob=TRUE, breaks="FD", plot=FALSE)   
    plot(myhist$mids, myhist$density, log="y", type="h", lwd=5, 
         lend=2, main="histogram: density on log scale")
    plot( function(x) dnorm(x,mean(ydata), sd(ydata)), 
          from=min(myhist$mids), to=max(myhist$mids), col="red", 
          lwd=3, add=TRUE)

Could be even more interesting if showing error bars on the histogram, see Confidence interval for the height of a histogram bar.

Another helpful tool for distribution identification is the Cullen and Frey graph implemented in fitdistrplus (R):

Cullen and Frey graph

This is quite interesting. The big blue point is the data shown on a squared skewness versus kurtosis plane, the yellow points are bootstrapped versions. All points are way away from the parts of the plane with named distributions, none of them have high enough kurtosis. The suggestion in another answer of looking into t distribution could be good. R code for last plot:

    library(fitdistrplus)
    descdist(ydata, boot=50)  
$\endgroup$
1
  • 1
    $\begingroup$ This is a great take on fitting! $\endgroup$ Commented Nov 7, 2018 at 12:29
0
$\begingroup$

Do you have any alternative candidate distributions? When fitting distributions, I've always considered multiple options, relevant to the studied system. This distribution $\textrm{ad oculum}$ looks, especially after looking at qq plots, like the Student's t to me, yet this is just an unverified observation.

$\endgroup$
1
  • 2
    $\begingroup$ Student $t$ distributions don't twist around the middle of the Normal probability plot like this one. Regardless, your initial advice gets to the point: namely, why should it matter that the data distribution conform to any known nice mathematical distribution at all? $\endgroup$
    – whuber
    Commented Nov 6, 2018 at 21:41
0
$\begingroup$

Considering $n=5800$, which is a huge sample, I'd not accept those qqplots. Your second qqplot suggests your smaller observations are too small to be from a normal distribution, and your larger observations are too big to be from a normal distribution. Your tail probabilities are too heavy for a normal distribution.

I don't know what the distribution is. Your data is probably some sort of ratio of random variables. Perhaps the distribution is Cauchy? or t?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.