Shapiro test only checks non-normality?

Question

I am a bit confused regarding the Shapiro (and Anderson-Darling) test. I have 2 datasets with about 100 columns each and would like to perform a t-test between columns (meaning column 1 in dataset 1 tested against column 1 in dataset 2 and so on).

I know that the t-test is robust but I still would like to test that the data is at least somewhat normally distributed.

So please correct me if I'm wrong but our null hypothesis for the Shapiro is that the data is normally distributed vs alternative hypothesis that the data is not.

So if I get a p-value that is $<0.05$ we reject the null hypothesis and my data is NOT normally distributed. If I get a p-value $> 0.05$ we do not reject the null hypothesis but that doesn't mean the data is normally distributed anyway right?

So how do I know if it is? Should I then always check the qq-plot if the p-value is $>0.05$? Or maybe the Shapiro test is not even used for this kind of thing?

I read this post but I'm still having a hard time understanding. If you have please also post sources since this is for my thesis.

Example of my data (from the first column in the first dataset) in R.

data1<-c(5.43, 5.58, 5.83, 5.76, 5.73, 6.02, 5.89, 6.21, 5.84, 5.58, 5.21, 5.96, 6.12, 5.91, 6.35, 6.35, 6.46, 6.41, 5.78, 5.96, 5.46, 5.91, 5.68, 5.80, 5.49, 5.50,
         5.83, 5.61, 6.18, 5.18, 6.53, 7.23, 6.21, 6.31, 6.86, 6.62, 6.95, 6.00, 6.29, 6.77, 5.25, 6.41, 6.43, 5.32, 6.17, 6.42, 6.19, 5.92, 6.20, 5.94, 5.38, 5.93,
         6.39, 5.79, 6.80, 5.68, 6.73, 6.68, 5.77, 5.95, 5.95, 6.48, 6.30, 6.15, 5.64, 5.21, 5.84, 5.90, 5.54, 6.59, 5.93, 6.48, 5.69, 6.52, 6.62, 5.78, 6.50, 6.68,
         5.35, 5.84, 6.08, 5.85, 5.91, 5.78, 6.52, 5.77, 6.25, 6.11, 6.64, 6.20, 5.45, 5.20, 6.41, 6.95, 6.45, 5.58, 6.54, 5.89, 5.21, 6.35, 5.73, 6.34, 6.61, 5.99,
         6.07, 6.48, 6.31, 6.92, 7.22, 7.90, 6.17, 5.97, 6.22, 6.00, 5.91, 5.88, 6.79, 6.50, 5.64, 6.02, 6.20, 5.74, 6.12, 5.47, 6.31, 6.58, 6.71, 7.35, 6.54, 5.89,
         5.98, 7.62, 6.48, 6.26, 6.51, 6.02, 6.06)

plot(density(data1))
shapiro.test(data1)
qqnorm(data1);qqline(data1, col = 2)

I think the density and qq plot says it is normally distributed but since I rejected the null hypothesis it is not?

Answer 1

By Shapiro test you may mean Shapiro-Wilk or you may mean Shapiro-Francia test. As this is not an R forum and questions and answers here need not, indeed usually should not, assume accessibility to, or use of, or knowledge of particular software, I just point out that ambiguity, presumably an ambiguity resolved by reading the documentation for shapiro.test in R (I decline the effort).

There are many, many posts on this forum cycling around the issues you raised. Your post should probably be closed as a duplicate, but some headlines are:

1. One needs to be clear why marginal normality is of interest in the first place. You answer this quite well by flagging an aim of applying $t$ tests.

2. The question Is the data normally distributed? cannot be answered Yes or No like a question of whether you own a passport or have a particular university degree, except that strictly, but not usefully, all empirical datasets fail to be exactly normal. Even using a specific test (e.g. Shapiro-Wilk) and a conventional $P$-value threshold is only part of the answer, as the focus should be whether there is a better description than "non-normal" that fits your purposes.

3. Your example variable (you say column) is an interesting case in point. A quantile-normal plot (one below) shows mild but definite skewness and a Shapiro-Wilk test (in Stata; your results may differ slightly depending on the $P$-value calculations) gives me a $P$-value of 0.0184. That means the result is definitely significant, so that we can say that the distribution is definitely not normal. Indeed the graph tells us that more informatively, as we can see in what way the distribution is not normal. What both theory and experience teach is that with a large enough dataset any departure from normality will qualify as significant and you have to check whether the departure is not just significant, but also important.

4. What the test cannot tell us is whether an assumption of normal distribution is good enough for your purposes. You've alluded to part of the answer by referring to the known robustness of the $t$ test. Another partial answer is to consider whether we might be better off with transforming the variable. Here with a small relative range (maximum 7.9/minimum 5.18 is about 1.53) the expectation is that no transformation will make much difference: with all values positive, any plausible transformation will be close to linear over a small range). However, a square root or log transformation will help a bit. My calculations give a skewness for the data you gave of 0.550, reduced a bit by taking square roots, and reduced a bit more to 0.277 by taking logarithms. Results for kurtosis show a parallel improvement towards normality.

If these were my data, I would consider a log transformation if and only if both variables (columns) had similar moderate skewness. Without any information on what these data are, it is impossible to comment on whether that is a good idea scientifically (economically, sociologically, whatever).

I always put more weight on what a quantile-normal plot tells me than on any formal goodness-of-fit test. This is genuinely difficult for beginners, as you need experience to interpret such plots confidently and only by doing it several times do you get that experience. But that is just like any other learning experience....