6
$\begingroup$

I came across a paper on predicting performance metrics for Facebook posts. The dataset is available on this Link

In that the authors make the statement below:

We adopted the Shapiro–Wilk test to assess if each of the output columns for the features to be predicted followed a normal distribution. Such a validation provided the ground needed to discard the 5% posts from which the performance metric value deviated the most, leaving 751 of the samples for building the model.

Does it mean: they apply Shapiro-Wilk test on the dependent variable and if does not pass the test then they discard certain samples from the data set?

Edit:

The paper can be downloaded from here

$\endgroup$
8
  • 2
    $\begingroup$ You should never delete observations that appear to be outliers so that the remaining data can pass a normality test such as the the Shapiro Wilk Test. $\endgroup$ Commented Jan 23, 2017 at 18:38
  • 2
    $\begingroup$ I never understood the logic of a test of the null hypothesis that a distribution is exactly normal when no actual distributions are exactly normal. $\endgroup$
    – David Lane
    Commented Jan 23, 2017 at 18:49
  • $\begingroup$ Can you provide a link to the full article (e.g. Google Drive, Box, Dropbox)? Readers may not have access to see the article. $\endgroup$
    – Jon
    Commented Jan 23, 2017 at 18:55
  • $\begingroup$ @Jon I have put a link to download the paper. $\endgroup$
    – cps
    Commented Jan 23, 2017 at 21:35
  • 1
    $\begingroup$ The paper can be downloaded from the link in edit section. $\endgroup$
    – cps
    Commented Jan 23, 2017 at 22:09

1 Answer 1

2
$\begingroup$

Does it mean: they apply Shapiro-Wilk test on the dependent variable and if does not pass the test then they discard certain samples from the data set?

No. It sounds like they did this for the features. Their methodology seems to follow this approach (not to say that I support their approach):

  1. Run a feature through the Shapiro-Wilk test for normality.

  2. Check the p-value of the test.

  3. If the p-value is high (say above $0.05$), do not modify the data.

  4. If the p-value is low (say below $0.05$), perform some kind of outlier screening to remove observations from the entire data set.

This is then repeated for all features, meaning that, if feature $1$ leads to the removal of observations $7$ and $62$ while feature $2$ leads to the removal of observations $9$, $62$, and $90$, then observations $7$, $9$, $62$, and $90$ are omitted from the analysis.

Two comments to the original question align with my take on this methodology.

You should never delete observations that appear to be outliers so that the remaining data can pass a normality test such as the the Shapiro Wilk Test.

-Michael R. Chernick

I never understood the logic of a test of the null hypothesis that a distribution is exactly normal when no actual distributions are exactly normal.

–David Lane

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.