I have noticed it is somewhat common for people to collect data, perform a statistical test (e.g., t-test) and then also look for possible "outliers" in the data. Then they remove the outliers and also perform the test on the filtered data.

If it is significant in both cases it is concluded that "the outliers don't affect the result". If it is significant only in the second case then usually this is noted and further analysis/discussion is done using only the "non-outliers". This seems to be considered acceptable as long as it is reported that you did it.

I am wondering if anyone has thought deeply about this and discussed the logic of doing so. Also, has anyone investigated this process to determine the consequences for false-positive rate and power function.

Edit: To clarify, when I am thinking of "outliers" these are data points that appear different from the rest but there is not necessarily a theoretical reason to treat them differently. The data itself suggests they are different in some unknown way. In that way this question may differ from previous questions on the site.

As an example of discussing the logic, Roger Ratcliff makes a well-reasoned argument in the case of reaction-time data:

The outliers' distribution most likely has a much greater standard deviation than the distribution of the real process, giving a long and elevated tail, because outliers often come from spurious events such as loss of attention, daydreaming, distraction, and so forth. Determining whether any particular group of extreme reaction times contains mostly real responses or mostly outliers is extremely difficult. Therefore, finding methods that minimize the effects of suspect observations is an important aim of statistical methods in the analysis of reaction time. In fact, the goal for our models and empirical research should be to account for the middle 85-95% of the observations in our reaction time distributions; these are the data that are most likely to come from the real processes under consideration and also most likely to be critical in testing hypotheses and models.

Psychol Bull. 1993 Nov;114(3):510-32. Methods for dealing with reaction time outliers. Ratcliff R. (pg 2 of paper)

I would agree with his assessment. But what if you do not have any a priori reason to suspect your data will contain outliers other than "individual variability"?

Edit 3: Something I thought of regarding "the outliers don't affect the result". Clearly the inclusion/exclusion of outliers will affect any parameter estimation even if the p-value is still on the same side of the significance threshold. So in that sense the claim is false.

Therefore it is not something that should be taken lightly (described in one sentence) because it will impede meta-analysis if different researchers use different definitions of outlier. Also, the exact values of the outliers would be of interest to future researchers since they may observe a similar phenomenon caused by the same process.

  • 2
    $\begingroup$ You may want to check this site for outliers. Quite a bit has been written and some should be meaningful to you.stats.stackexchange.com/questions/16709/… $\endgroup$ – doug.numbers Dec 16 '13 at 16:10
  • $\begingroup$ @doug.numbers Thanks, missed that one. The paper cited looks highly relevant. $\endgroup$ – Flask Dec 16 '13 at 16:42
  • $\begingroup$ The properties of hypothesis tests will be impacted by such conditional approaches. There are a number of references for a variety of situations given here, though none exactly correspond to the situation in your question. $\endgroup$ – Glen_b Dec 16 '13 at 20:16
  • $\begingroup$ @Glen_b Thanks. I was more wondering if this behavior has been introduced/examined or is just kind of a custom that has grown up around significance testing without much deep thought. Checking the data multiple ways would definitely increase the chance of finding significance. $\endgroup$ – Flask Dec 18 '13 at 1:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.