I m trying to check my data normal_data for normal distribution in R.
I always used shapiro.test or ks.test, but now I have more than 5000 values to check.
Is there any other function or possibility to check the data?
$\begingroup$
$\endgroup$
8
-
2$\begingroup$ Why are you doing this? Why do you need to check this? $\endgroup$– RolandCommented Oct 29, 2021 at 17:47
-
$\begingroup$ ... and why is 5000 a critical value? $\endgroup$– LimeyCommented Oct 29, 2021 at 18:00
-
4$\begingroup$ @Limey Most software does not bother to compute the sampling distribution (or its critical values) for distributional tests of large sample sizes, because (a) it can be a lot of work and (b) is pointless. $\endgroup$– whuber ♦Commented Oct 29, 2021 at 18:06
-
3$\begingroup$ @Tar it is incorrect to conclude that "$n\gt 30$" is "large enough." It is also incorrect to conclude one can therefore "ignore the distribution and use parametric tests." The CLT does not assert what you claim it does. Exactly whether and how to check one's data depends on what kinds of tests one plans on making. $\endgroup$– whuber ♦Commented Oct 29, 2021 at 18:20
-
3$\begingroup$ @Dave The possible misunderstanding, now deleted, was subtler than that. It read "According to the central limit theorem, no matter what kind of distribution we have, the sampling distribution tends to be normal if the sample is large enough (n $\gt$ 30)." Missing from this was any sense of what statistic is involved. For instance, the sampling distribution of the maximum is not going to be Normal (and rarely even close to it). $\endgroup$– whuber ♦Commented Oct 29, 2021 at 19:27
|
Show 3 more comments
2 Answers
$\begingroup$
$\endgroup$
For the question in the title, see How to perform a test using R to see if data follows normal distribution which gives many possibilities. As for why the limitations of the Shapiro-Wilk test, see Can a sample larger than 5,000 data points be tested for normality using shapiro.test by applying the test to a subsample?
Then, (as mentioned by many in comments), why do you do this? See Testing large dataset for normality - how and is it reliable? and Is normality testing 'essentially useless'?
answered Oct 30, 2021 at 14:06
$\begingroup$
$\endgroup$
For example, you could use Jarque Bera test, see there. But, in almost all cases you should reject the null hypothesis despite that you see bell-shaped distribution of your data. The reason is that usual tests are quite sensitive to the big samples. Solution of this problem is using qq plot and histograms with choosing certain bins. Also, read this useful discussion.
