# Shapiro Wilk Test

I came across a paper on predicting performance metrics for Facebook posts. The dataset is available on this Link

In that the authors make the statement below:

We adopted the Shapiro–Wilk test to assess if each of the output columns for the features to be predicted followed a normal distribution. Such a validation provided the ground needed to discard the 5% posts from which the performance metric value deviated the most, leaving 751 of the samples for building the model.

Does it mean: they apply Shapiro-Wilk test on the dependent variable and if does not pass the test then they discard certain samples from the data set?

Edit:

• You should never delete observations that appear to be outliers so that the remaining data can pass a normality test such as the the Shapiro Wilk Test. Commented Jan 23, 2017 at 18:38
• I never understood the logic of a test of the null hypothesis that a distribution is exactly normal when no actual distributions are exactly normal. Commented Jan 23, 2017 at 18:49
– Jon
Commented Jan 23, 2017 at 18:55
– cps
Commented Jan 23, 2017 at 21:35
– cps
Commented Jan 23, 2017 at 22:09

Does it mean: they apply Shapiro-Wilk test on the dependent variable and if does not pass the test then they discard certain samples from the data set?

No. It sounds like they did this for the features. Their methodology seems to follow this approach (not to say that I support their approach):

1. Run a feature through the Shapiro-Wilk test for normality.

2. Check the p-value of the test.

3. If the p-value is high (say above $$0.05$$), do not modify the data.

4. If the p-value is low (say below $$0.05$$), perform some kind of outlier screening to remove observations from the entire data set.

This is then repeated for all features, meaning that, if feature $$1$$ leads to the removal of observations $$7$$ and $$62$$ while feature $$2$$ leads to the removal of observations $$9$$, $$62$$, and $$90$$, then observations $$7$$, $$9$$, $$62$$, and $$90$$ are omitted from the analysis.

Two comments to the original question align with my take on this methodology.

You should never delete observations that appear to be outliers so that the remaining data can pass a normality test such as the the Shapiro Wilk Test.

-Michael R. Chernick

I never understood the logic of a test of the null hypothesis that a distribution is exactly normal when no actual distributions are exactly normal.

–David Lane