I have noticed it is somewhat common for people to collect data, perform a statistical test (e.g., t-test) and then also look for possible "outliers" in the data. Then they remove the outliers and also perform the test on the filtered data.
If it is significant in both cases it is concluded that "the outliers don't affect the result". If it is significant only in the second case then usually this is noted and further analysis/discussion is done using only the "non-outliers". This seems to be considered acceptable as long as it is reported that you did it.
I am wondering if anyone has thought deeply about this and discussed the logic of doing so. Also, has anyone investigated this process to determine the consequences for false-positive rate and power function.
Edit: To clarify, when I am thinking of "outliers" these are data points that appear different from the rest but there is not necessarily a theoretical reason to treat them differently. The data itself suggests they are different in some unknown way. In that way this question may differ from previous questions on the site.
As an example of discussing the logic, Roger Ratcliff makes a well-reasoned argument in the case of reaction-time data:
The outliers' distribution most likely has a much greater standard deviation than the distribution of the real process, giving a long and elevated tail, because outliers often come from spurious events such as loss of attention, daydreaming, distraction, and so forth. Determining whether any particular group of extreme reaction times contains mostly real responses or mostly outliers is extremely difficult. Therefore, finding methods that minimize the effects of suspect observations is an important aim of statistical methods in the analysis of reaction time. In fact, the goal for our models and empirical research should be to account for the middle 85-95% of the observations in our reaction time distributions; these are the data that are most likely to come from the real processes under consideration and also most likely to be critical in testing hypotheses and models.
Psychol Bull. 1993 Nov;114(3):510-32. Methods for dealing with reaction time outliers. Ratcliff R. (pg 2 of paper)
I would agree with his assessment. But what if you do not have any a priori reason to suspect your data will contain outliers other than "individual variability"?
Edit 3: Something I thought of regarding "the outliers don't affect the result". Clearly the inclusion/exclusion of outliers will affect any parameter estimation even if the p-value is still on the same side of the significance threshold. So in that sense the claim is false.
Therefore it is not something that should be taken lightly (described in one sentence) because it will impede meta-analysis if different researchers use different definitions of outlier. Also, the exact values of the outliers would be of interest to future researchers since they may observe a similar phenomenon caused by the same process.