3
$\begingroup$

I have got the following dataset:

Time             Value                                                                 
2018-08-01 00:00,4.3
2018-08-02 00:00,3.3
2018-08-03 00:00,5.3
2018-08-04 00:00,6.3
2018-08-05 00:00,3.3
2018-08-06 00:00,3.3
2018-08-07 00:00,1.3
2018-08-08 00:00,2.3
2018-08-09 00:00,4.3
2018-08-10 00:00,3.3
2018-08-11 00:00,5.3
2018-08-12 00:00,6.3
2018-08-13 00:00,3.3
2018-08-14 00:00,3.3
2018-08-15 00:00,1.3
2018-08-16 00:00,2.3
2018-08-17 00:00,4.3
2018-08-18 00:00,3.3
2018-08-19 00:00,5.3
2018-08-20 00:00,6.3
2018-08-21 00:00,3.3
2018-08-22 00:00,1.3
2018-08-22 00:00,4.3
2018-08-23 00:00,2.3
2018-08-24 00:00,4.3
2018-08-25 00:00,3.3
2018-08-26 00:00,5.3
2018-08-27 00:00,6.3
2018-08-28 00:00,3.3
2018-08-29 00:00,3.3
2018-08-30 00:00,1.3
2018-08-31 00:00,2.3
2018-09-01 00:00,4.3
2018-09-02 00:00,3.3
2018-09-03 00:00,2.3
2018-09-04 00:00,6.3
2018-09-05 00:00,3.3
2018-09-06 00:00,3.3
2018-09-07 00:00,3.3
2018-09-08 00:00,2.3
2018-09-09 00:00,4.3
2018-09-10 00:00,3.3
2018-09-11 00:00,5.3
2018-09-12 00:00,4.3
2018-09-13 00:00,3.3
2018-09-14 00:00,3.3
2018-09-15 00:00,4.3
2018-09-16 00:00,2.3
2018-09-17 00:00,6.3
2018-09-18 00:00,5.3
2018-09-19 00:00,5.3
2018-09-20 00:00,6.3
2018-09-21 00:00,6.3
2018-09-22 00:00,3.3
2018-09-22 00:00,4.3
2018-09-23 00:00,2.3
2018-09-24 00:00,4.3
2018-09-25 00:00,3.3
2018-09-26 00:00,5.3
2018-09-27 00:00,6.3
2018-09-28 00:00,3.3
2018-09-29 00:00,4.3
2018-09-30 00:00,2.3
2018-10-01 00:00,4.3
2018-10-02 00:00,3.3
2018-10-03 00:00,5.3
2018-10-04 00:00,6.3
2018-10-05 00:00,1.3
2018-10-06 00:00,3.3
2018-10-07 00:00,3.1
2018-10-08 00:00,2.3
2018-10-09 00:00,4.3
2018-10-10 00:00,3.3
2018-10-11 00:00,5.3
2018-10-12 00:00,6.3
2018-10-13 00:00,3.3
2018-10-14 00:00,3.3
2018-10-15 00:00,4.3
2018-10-16 00:00,2.3
2018-10-17 00:00,4.3
2018-10-18 00:00,3.3
2018-10-19 00:00,5.3
2018-10-20 00:00,6.3
2018-10-21 00:00,7.2
2018-10-22 00:00,3.3
2018-10-22 00:00,4.3
2018-10-23 00:00,2.3
2018-10-24 00:00,4.1
2018-10-25 00:00,3.3
2018-10-26 00:00,5.9
2018-10-27 00:00,6.3
2018-10-28 00:00,4.1
2018-10-29 00:00,3.1
2018-10-30 00:00,4.3
2018-10-31 00:00,2.3

I applied the Shapiro_Wilk_Normality_Test and Kolmogorov_Smirnov_Test with alpha = 0.05. As result, I got

  1. Shapiro_Wilk_Normality_Test: Statistics=0.933,p=0.000
  2. Kolmogorov_Smirnov_Test:Statistics=0.937, p=0.000

In both cases, the sample does not look Gaussian (reject $H_0$). Then I applied DAgostino_K_2_Test

  1. DAgostino_K_2_Test Statistics=4.230, p=0.121 which would make to think of Sample looks Gaussian (fail to reject H0).

Then, I visualized the data with pyplot.hist and it looked like Gaussian.

My question is: the data seems to have a Gaussian, and D'Agostino test seems to figure it out whereas Shapiro-Wilk and Kolmogorov-Smirnov seem to reject it. Any particular explanation? It would be very useful.

Improve this question
$\endgroup$
2
  • $\begingroup$ Visually inspect your data with a qqplot. Look at the behavior at the ends. I don't know about the difference in these normality tests to answer your desired question but this can guide you in assessing your data beyond just a histogram. $\endgroup$
    – mindhabits
    Feb 25, 2019 at 17:34
  • $\begingroup$ The histogram failed to materialise. That may be a side-effect of your reputation. However, a histogram would almost certainly hide the discreteness which is, or should be, central to any answer. $\endgroup$
    – Nick Cox
    Feb 25, 2019 at 18:25

2 Answers 2

Reset to default
1
$\begingroup$

Your data are a series of spikes. In a very loose way, they are approximately normal if you ignore that. Some tests are sensitive to such discreteness and some are not. Despite the extraordinary menagerie of tests that people keep devising, well-chosen graphs show these features clearly and many tests do not.

Most have 0.3 as fractional part. There should be a story behind that.

enter image description here

The real question is: why you think it's important whether they are normally distributed? What analyses do you intend where it's important to have normally distributed data?

Improve this answer
$\endgroup$
4
  • $\begingroup$ Ni@ck Cox: Thank you very much for your answer. Because they are series of spikes I have got that different tests may provided different results, depending on its sensitivity (see Shapiro_Wilk_Normality vs DAgostini test). The answer to your question is to decide whether I can apply a parametric approach or non-parametric approach to detects novelties. $\endgroup$
    – Carlo Allocca
    Feb 25, 2019 at 18:47
  • $\begingroup$ I am not sure that that description allows much expansion of any answer. What would be a novelty be? Again, why are almost all values integer.3? $\endgroup$
    – Nick Cox
    Feb 25, 2019 at 19:24
  • $\begingroup$ Thanks Nick. Sorry for replying so late, I was just unwell. Novelty has the same meaning of the anomaly. The 0.3 is because of some rounding of the device. $\endgroup$
    – Carlo Allocca
    Feb 28, 2019 at 17:03
  • $\begingroup$ Still can't add further. There is some context here ("a device") which could be crucial but nevertheless remains hidden. Simply, any methods assuming a normal distribution might work fine or badly depending on what they are. $\endgroup$
    – Nick Cox
    Feb 28, 2019 at 18:16
1
$\begingroup$

I see some normality in this dataset, but I'd be more inclined to call it bimodal, with about 75% of the data centered at 3.5 and 25% centered at 6.

As for the differences between the test results: The Shapiro-Wilk test doesn't perform well when there are many repeat values, which we see in this dataset. (It's very strange, actually. 94% of the values end with a three in the tenths place.) The D'Agostino test evaluates the distribution using the skewness and kurtosis, which are both fairly low here (0.24 and -0.66, respectively).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you. it is very good to know that Shapiro-Wilk test doesn't perform well when there are many repeat values. Actually, I should not use it then. However, it seems that in this specific case produce the same results of Kolmogorov-Smirnov. Please could you tell how to use properly the D'Agostino test to reject H0, in this case? Code sample? $\endgroup$
    – Carlo Allocca
    Feb 25, 2019 at 18:55
  • $\begingroup$ The p-value you obtained from the D'Agostino test and your interpretation are both accurate. What sort of code sample are you looking for and for what program? $\endgroup$
    – Courtney
    Feb 25, 2019 at 19:39
  • $\begingroup$ Thank you Courtney. Sorry for replying late, I was just unwell. If everything seems to be good, we got the Shapiro-Wilk and Kolmogorov-Smirnov implies NO and D'Agostino implies YES. I was thinking of python code $\endgroup$
    – Carlo Allocca
    Feb 28, 2019 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.