10
$\begingroup$

I am doing a thesis and I am stuck with my statistics on time series data. And when I run a Shapiro test, or any test, it says:

Statistics=0.998, p=0.000
Sample does not look Gaussian (reject H0)

And that it is not normally distributed. But when I look to my QQ plot and histogram the data looks normally distributed. And even when it would not pass the test, it is strange that it results in p = 0.000.

I also have two other groups where the QQ plot and histogram looks way worse than this one, but the tests say those are normally distributed.

I am looking in my script, but I can’t find any mistakes so far. Is it likely that there is something wrong or is it very normal for this data to be nonparametric (even with this graphs)?

I am using Python btw, not R.

QQ plot:

QQ plot

Histogram:

Histogram

PS. I also have a box plot:

Enter image description here

$\endgroup$
7
  • 21
    $\begingroup$ You seem to have a huge sample size. Normality tests will detect even trivial deviations from normality if your sample size is large enough (discussed multiple times here at CV, take a look at this search) - and your q-q plot shows clear departures from normality in the tails. I suspect the more important question is why you are interested in the normality of your data in the first place. $\endgroup$ Jun 23 at 8:27
  • 1
    $\begingroup$ Thanks for your response, I have did random samples to reduce the amount, and I did several test which are ment for large datasets. And still the plots look very normally distributed. And the other groups are normally distributed with way worse results. And if they are normally distributed I want them to be normally distributed. $\endgroup$
    – SimonDL
    Jun 23 at 8:44
  • 4
    $\begingroup$ See this excellent post: stats.stackexchange.com/questions/2492/… $\endgroup$ Jun 23 at 12:04
  • 3
    $\begingroup$ Note that the normality test will not tell you "this data is normally distributed" - you can only fail to reject a null hypothesis, not confirm it. The test can only tell you "there is insufficient evidence to conclude this data is not normally distributed". That can happen either because 1) the data is indeed normally distributed, or 2) the data is not normally distributed but you don't have a large enough sample size to prove it. A failure to reject H0 may or may not be evidence of normality depending on the sample size/power of the test. $\endgroup$ Jun 23 at 16:16
  • 2
    $\begingroup$ What is on the x-axis in the second plot? What is the bin size? $\endgroup$ Jun 23 at 17:22

4 Answers 4

8
$\begingroup$

Ben has already explained much.

If you only want some kind of classifier based on, say, variance, then you could use a common variance estimator and you don't need normality. Your distributions should have second moments, though.

You wrote that you did "many tests" - this is rather suspicious. Are you aware of the multiple hypothesis testing issues? Are you aware of the "formulating hypotheses on the data" issues? I got the impression, that you'd prefer estimators, not hypothesis testing. Your statistical work seems to have some basic flaws...

$\endgroup$
1
  • 2
    $\begingroup$ Multiple hypothesis testing is always a valid concern, but there's not THAT many normality tests. OP's p value is <10^-3, and I doubt he literally tried hundreds of normality tests, so the FDR correction wouldn't alter the results much in this case. $\endgroup$
    – Jessica
    Jun 24 at 22:39
27
$\begingroup$

You QQ plot shows that there is departure from normality in the tails of your distribution. Since you have a very large sample size, your test is able to detect even a relatively minor departure from normality, so it is not surprising that it is rejecting the null hypothesis with a very low p-value. Your test is correctly identifying that your data do not come from a normal distribution (or their tail behaviour would not look like that).

Incidentally, this is one of the properties of a classical hypothesis test when you use it to test a narrow hypothesis such as this one --- if the hypothesis is even slightly false then the p-value will converge to zero as the sample size becomes large. Statisticians are therefore cautious when we use hypothesis tests and we are wary of using them to test very narrow hypotheses. If the null hypothesis is too narrow then it will almost certainly be false, and the test will detect that once you give it enough data.

$\endgroup$
2
  • 1
    $\begingroup$ Thank you so much for your response Ben. You expressed it very clear, I have trouble with that. So I hoped it was normally distributed so I could use a standard deviation so I can classify my data. So the std of my three categories would be able of getting a predicted label. This is some kind of very basic machine learning. And using the STD I get around a 95% accuracy, but I do not think I can use a simple python std function if my data is not devided, as it will not use the mean at the right place. $\endgroup$
    – SimonDL
    Jun 23 at 21:27
  • $\begingroup$ See also: people using zero correlation for their null hypothesis. Negligible? Sure. Zero? $\endgroup$
    – TLW
    Jun 24 at 6:23
16
$\begingroup$

No data will ever be perfectly normally distributed, so it is not surprising that your data are "not normal", and note that no test ever will say "data are normal"; the best any test can tell you is that according to the test statistic data do not deviate from what is expected under the normal distribution, but data may not be normal anyway (for example due to not being truly continuous, as no data are).

As data are never truly normal, true normality is not what is of interest in applied data analysis. What is really relevant is if you have deviations from normality that have the potential to mislead conclusions of methods that supposedly assume normality. But this depends on what you think you need normality for, i.e., what you want to do with the data after having tested normality. Chances are the data shown in your plots are fine for most things you'd like to do (there are no extreme outliers and no strong skewness, so theory based on normality will give very good approximations), but without knowing what you want to do based on normality, we can't know for sure. (Note that most analyses have further assumptions on top of normality, and sometimes problems with those are worse, particularly independence.)

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your response! Yes I think my data looks very proper this way, I reacted to Ben what I really wanted to achieve. To use the std to classify predicted/train data. And it works well with the std, I am not sure if I am allowed to do this. $\endgroup$
    – SimonDL
    Jun 23 at 21:29
  • 1
    $\begingroup$ @SimonDL I'm not sure what you mean by "use the std to classify", however there is nobody who can forbid this. We can't tell whether it's the best possibility in your situation, but you can use cross-validation to find out how well you're doing. $\endgroup$ Jun 23 at 22:29
7
$\begingroup$

Why is my data not normally distributed while I have an almost perfect QQ plot and histogram?

Because it is not perfect

Your qq plot is not perfect. You have some clear deviations from a normal distribution that are statistically significant.

OP's image with highlights in the tails of the distribution showing deviations

Normal distributions exist in mathematics but not in nature

These type of deviations occur a lot. The Normal distribution occurs as a limit distribution for a sum of many different sources of variation. See for instance the central limit theorem. But this limit occurs at infinity and that is almost never exactly obtained in practice (although with physical experiments, which can have extremely large numbers of sources of variation, it can get extremely close).

Potential causes for your difference

In your case there is a deviation from normality in the limits of the distribution. Your sample has a limit between -8.75 and -6.5. On the other hand a normal distribution has limits $\pm \infty$.

  • Such bounds can occur because you do not have a large sum of variables with variations. For instance the sum of 100 rolls with a six sided dice can be well approximated by a normal distribution. But, the dice rolls are limited in the range from 0 to 600.

  • Another possibility: it is very likely that there might be some theoretical limit.

    Take for instance the performance of a 100 meter sprinter. This might be very well approximated with a normal distribution since there are many sources of error involved in the performance (reaction time at the start, wind speed, physical state, positioning the feet correct every step, etc.). But, you will never get a human running faster than 0 seconds, whereas a normal distribution includes this as an option. Before these 0 seconds there will be some other limit. The sources of error mostly add linearly but once the runner obtains a certain speed then wind resistance starts to increase, coordination becomes more difficult and many other factors start to play a role. The addition of the error is not linear and independent (which means that the normal distribution approximation works less well)

It seems to me that it is physically impossible for your variable to be normal distributed, and you do not need a statistical test for this (I imagine that sessility, although I am not sure how you measure this and get negative values, is bounded).

Practical solution

Very often you do not need exact normality.

(edit: as Christian Hennig mentions in the comments, this is too liberal, it should be more strict instead and the statement should be "You never need exact normality". This relates to 'all models are wrong'. The assumption of normality is a part of a model that is likely (or certainly) to be false, but the point of modeling with the assumption of a normal distribution is that it is close enough.)

More important than significance is the effect size.

Although a problem here is that there is not really a good rule of thumb in the case of normality tests. Depending on your goals/case there might be recommendations. For instance, the qq-plot shows that you variable has smaller tails. This means that your estimate is gonna have a smaller excessive variability. Then you do not have to worry about underestimating p-values and increased type I errors (if the variability of your estimate is less, then you are less likely to exceed a certain range of statistical significance).

$\endgroup$
4
  • $\begingroup$ Well thank you for your great response! Well I have made the data logarithmic, this because it is based on a absolute gradient. A way to solve this my supervisor told me to make it logaritmic. $\endgroup$
    – SimonDL
    Jun 24 at 8:42
  • $\begingroup$ A related question about normality of log transformed variables is here: stats.stackexchange.com/questions/93082/… $\endgroup$ Jun 24 at 8:51
  • $\begingroup$ "Very often you do not need exact normality." - If any method needs exact normality, it can never be applied, as exact normality doesn't exist in practice. $\endgroup$ Jun 24 at 10:49
  • 1
    $\begingroup$ @ChristianHennig that is a good point. I was too cautious. The mathematical formulations assume normality, but we do not really have that normality. It is not about whether we have that normality but instead about how much different we are from normality. However, the question 'how much is too much', that is a difficult question. $\endgroup$ Jun 24 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.