Should outlier detected before or after train test split

Question

Outliers are usually first detected using Boxplot, then the suspicious observations may be sent to experts for justification - justify whether they are true outliers (contaminated data) or leverage points.
Suppose I need to perform model selection in a cross validation fashion. Should the outlier detection and justification done before or after data split?
If it should be done within cross validation, does it mean that in each round of cross-validation, I need to ask the experts to justify the outliers?

Answer 1

IMHO, both is possible.

• You may say that you work entirely on a clean data set, i.e. all data you admit has been checked beforehand to not contain outliers.

  • E.g. some instruments I work with have a measurement mode where the software already checks for particular types of outliers and automatically replaces the suspicious measurement by a fresh one. This would be clear from the reported measurement parameters/SOP.
  • IMHO it is also OK to clean your data set manually as described as long as you clearly state this limitation and as long as it is an acceptable limitation for the task at hand.
    Personally, I consider this the counterpart to ending the analysis/model development with an internal validation/verification (e.g. test data split off at the beginning) rather than a full validation study or external validation (by someone outside your organization).

• Doing the outlier detection inside cross validation explicitly makes it part of your model, like any other pre-processing step that applies a quality filter.
  In this case, consider setting up your outlier detection as a completely automatic sub-model and treating the expert opinion as reference for verfication/validation of your outlier-filter sub-model.

• There's a 2nd decision hidden here: do you exclude outliers only from training but not from prediction, or do you exclude them from both?

  IMHO almost everything is allowed during training - but you absolutely need to perform an honest verification/validation.
  To me, it is thus acceptable to exclude suspicious data or confirmed outliers from training if you think this will overall result in a better model. But unless you can argue that such points will be caught by a quality filter for application use, you need to assess your model's predictions on these data points. I then usually report performance in detail: overall performance, fraction of bad data points, performance for good data and performance for outliers.

• In any case, it is extremely important to be clear how this cleaning procedure affecty your model and predictions. In particular, whether the cleaning procedure has the potential to create an artificially easy problem.

---

does it mean that in each round of cross-validation, I need to ask the experts to justify the outliers?

I assume the expert is giving their judgment for each data point separately without looking at other.

In that case, I'd treat this as another type of reference label which is unknown for all data at the beginning and updated on the fly as suspicion arises. That is, when a data point comes up as suspicious, you look up that reference information. If its status is known already (i.e. the expert has seen it already) you proceed on this information, if its unknown, ask the expert and update the reference information.

(You may also decide to always ask the expert and then have a look at how consistent their judgment is. Particularly, if the type of outlier you're looking at is not very obvious)

---

IMHO asking expert opinion from an application point of view makes sense mostly iff:

• the opinion is needed for training, but no outlier filter is considered for application use
• the opinion is needed for verification/validation of the outlier filter, and during application use any data identified as suspicious by this filter is marked as "suspicious" as part of the prediction.

In other words, if expert opinion is part of your predicition procedure, application use will require an expert being around. This would often be considered an unacceptable limitation.