2
$\begingroup$

I am doing statistical analysis in a df and I am trying to check the normality of the data. I did the Shapiro Wilk test. I think I have to check if the data meets some assumptions, this is the case in the t-test. Basically I am trying through research to identify which are the assumptions that the data has to satisfy. I am new in statistics so any help will be appreciated ..thank you

$\endgroup$
5
  • $\begingroup$ Don't ever bother checking normality of the data. It is terrible practice. The t-test works by the CLT without normally distributed variables as long as they are iid. $\endgroup$
    – user327671
    Aug 9 at 5:50
  • $\begingroup$ Why are you checking "normality of the data"? I'm not aware of any statistical test that assumes normality of data. At most, a conditional normal distribution is assumed. $\endgroup$
    – Roland
    Aug 9 at 6:37
  • 1
    $\begingroup$ i did density plots , qq-plots and the shapiro wilk test in order to back up if the distribution is most likely normal or not because I have NAs and I want to replace them with the mean or the median , depend on if the distribution is normal or not but I am not sure if this is a correct approach@Roland $\endgroup$
    – pipts
    Aug 9 at 6:41
  • 1
    $\begingroup$ "I want to replace them with the mean or the median" Mean imputation distorts the statistical properties of your data. It should be avoided if you intend to use the imputed data for statistical tests. Look into multiple imputation approaches. $\endgroup$
    – Roland
    Aug 9 at 6:53
  • $\begingroup$ The t-test and many other methods can be affected by non-normality of the wrong kind, e.g., gross outliers. See here for a more general take on model assumption checking: stats.stackexchange.com/questions/538561/… $\endgroup$ Aug 9 at 9:14
1
$\begingroup$

You don't say how many observations you have. For small $n$ the Shapiro-Wilk test can fail to reject data sampled from non-normal populations. [Using R.] Neither test below has a P-value below $0.05 = 5\%.$

set.seed(808)
shapiro.test(rbeta(10, 2, 2))$p.val
[1] 0.8723119
shapiro.test(rgamma(15, 4, .2))$p.val
[1] 0.4677453

For small $n,$ it is probably best to assume that markedly asymmetrical samples, especially ones with extreme outliers in one tail, are not normal.

For very large $n,$ the S-W test may reject because of unimportant quirks in a few observations. And of course, the test will reject at the 5% level in 5% of samples that randomly sampled from normal populations. After several tries, I got the following result for data from a t distribution with 100 degrees of freedom, which is normal for most practical cases. [The implementation of the S-W test in R will not accept samples of size greater than 5000.]

shapiro.test(rt(5000, 100))$p.value
[1] 0.01595207

Nevertheless, I think it is worthwhile to look at normal probability plots (for distinctively nonlinear patterns), boxplots (for skewness or extreme outliers), and S-W tests (for P-values near $0)$ before performing a procedure that may give inaccurate results with non-normal data. The exponential sample of size $n = 35$ shown below would not be mistaken for a normal sample by any of these methods.

set.seed(1234)
x = rexp(35, .2)
shapiro.test(x)$p.val
[1] 0.001157203

enter image description here

par(mfrow = c(1,2))
 boxplot(x, col="skyblue2", pch=20)
 plot(qqnorm(x)); qqline(x, col="blue")
par(mfrow = c(1,1))

Notes: (1) A sample of size $n=35$ from an exponential population with $\mu = 5$ will almost always be rejected by the S-W test:

pv = replicate(10^5, shapiro.test(rexp(36,.2))$p.val)
mean(pv <= .05)
[1] 0.9892

Moreover, if one uses a t test for $H_0: \mu=5$ against $H_a: \mu \ne 5,$ then the rejection rate for a test intended to be at the 5% level will be closer to 7% than to 5%.

pv = replicate(10^5, t.test(rexp(36,.2), mu=5)$p.val)
mean(pv <= .05)
[1] 0.06892

(2) By contrast, the correct distribution theory with $\bar X/\mu \sim \mathsf{Gamma}(35,35)$ leads to a test that rejects with exactly the intended significance level.

An exact 95% confidence level for the sample x above is $(3.27, 6.36):$

mean(x)/qgamma(c(.975,.025), 35,35)
[1] 3.265594 6.364287

(3) While none of the links under "Related" in the margin of this page is an exact duplicate of your question, you might get useful insights by looking at them.

$\endgroup$
2
  • 1
    $\begingroup$ I did the shapiro test and qq plot. You suggest to perform and box plot to check about the skewness? @BruceET $\endgroup$
    – pipts
    Aug 9 at 7:06
  • $\begingroup$ Yes skewness would usually show in a boxplot, which would also show extreme outliers. // I added some notes at the end. $\endgroup$
    – BruceET
    Aug 9 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.