Removing outliers at the start when there are multiple ANOVA and correlational analyses in a single results section I would be grateful for opinion on which of the two options below (or an alternative) is best:
Summary of study: In a single results section, different ANOVAs are run on the different metrics – raw scores (such as RT, d prime, accuracy) and also collapsed/composite/index scores (for example combining RT and d prime, or normalised scores). Then these behavioural measures are correlated with survey data measures, and finally multiple regression analysis is run with behavioural and survey data. Hypotheses pertain separately to the behavioural analyses and the behavioural/survey correlational analyses.
Definitions:
Genuine outliers = participants with below chance-level accuracy, d-prime scores >3SDs below condition mean, participants not following instructions/technical issues
Outlying values = data points simply >3SDs away from condition mean
Option 1: Remove genuine outliers at the start, but note outlying values in analyses on a per analysis basis. For example:
Remove genuine outliers from whole dataset.
Then run each analysis for RT, d prime, normalised scores, correlations etc.
In each case (separately for each analysis), note if there are outliers (e.g. > 3SDs from condition means).
If data normally distributed:
Run the analysis with the outliers in.
Run the analysis with the outliers removed.
If results and assumptions not affected
Write up with outliers in – to keep the same N across analyses (and note outliers and the above analyses)
If data not normally distributed:
Remove outliers
if data now normally distributed:
Run analysis with outliers removed
Run non-parametric analysis with outliers included
If results are the same:
Write up non-parametric test with outliers in (keeps same N for all analyses) (note outliers and all above analyses)
If data not normally distributed:
Remove outliers
if data still not normally distributed
Run non-parametric analysis with outliers included
If results are the same:
Write up non-parametric test with outliers in (keeps same N for all analyses) (note outliers and all above analyses)
If a participant did not complete a survey, however, run the correlational analyses and regression with this smaller N.
Option 2: Remove all outliers and outlying values at the start. For example:
Remove genuine outliers from whole dataset
Then check all raw scores, collapsed score/indices, and survey measures for outliers (> 3 SDs from condition means)
Remove all these outliers, including participants that did not complete one or more of the survey measures (i.e. losing their behavioural data)
Run each of the analyses with this N
If there are new outliers and/or data not normally distributed:
Run parametric tests only
My concern with Option 2 is that there are no checks to determine whether these outlying values (kept in or removed) affect the findings. Also participants with no outlying values in raw scores are removed for having outlying values in collapsed/composite scores from all analyses. Plus behavioural data is removed due to missing survey data from purely behavioural analyses.
However, to remove all these outlying values (participants) at the start, and then to test whether the removal of each one affects results for each of the separate analyses (and also with the outliers in various combinations) requires a huge number of analyses!
Would be grateful for any suggestions or improvements to Options 1 or 2.
Thank you!
 A: More typically outliers are defined as near or mild outliers and far or extreme outliers. From nist Engineering statistics handbook 7.1.6. What are outliers in the data?
"Box plots with fences
A box plot is constructed by drawing a box between the upper and lower quartiles with a solid line drawn across the box to locate the median. The following quantities (called fences) are needed for identifying extreme values in the tails of the distribution:
lower inner fence: Q1 - 1.5*IQ
upper inner fence: Q3 + 1.5*IQ
lower outer fence: Q1 - 3*IQ
upper outer fence: Q3 + 3*IQ 

Outlier detection criteria
A point beyond an inner fence on either side is considered a mild outlier. A point beyond an outer fence is considered an extreme outlier."
Now as to what this means and why it does, consider the following. First, we do not know what type of distribution, e.g., skewed normal, log-normal, etc. we have in our data. Consequently, standard deviation and other normal associated distance measurements are not as good a yardstick to use as general rules as quantiles, which are more non-parametric. Thus, it is a more general rule to use quantiles as above, to classify outliers.
Now, as a general fule of thumb, one can expect near, mild outliers and such should be noted as an indication of possible non-normality. Far outliers I consider worthy of further investigation. For example, they may occur in very heavy tailed distributions, like the Cauchy distribution, in which case those far outliers may be a typical occurrence, or they may be due to a measurement system problem in which case they would be true outliers. I would not eliminate any outliers unless they are highly suspicious for being due to measurement system problems. For example, if extreme outliers are a result of the measurement system itself, I would model the system according to the needs of the measurement system, and not discard anything without good motivation. For example, for Cauchy distributed rv's one can use 25% truncated mean values, which are better than the median for determining the Cauchy peak value, but that is not the same thing as discarding outliers, per se.
Now this answer is not a direct answer to the questions posed, because I wouldn't do either of the choices offered.
Also see Handling outliers in ANOVA, Boxplot Outliers and
Simple algorithm for online outlier detection of a generic time series among others.
Also see footnote 13 of Comparison of the gamma-Pareto convolution with conventional methods of characterising metformin pharmacokinetics in dogs

As well as the treatment of outliers generated during bootstrap in that publication's Appendix Subsection entitled "Parameter errors from model-based bootstrap," which although not perfect explains a lot.
