2
$\begingroup$

I have been trying to understand the concepts of data leakage and outlier analysis as I am new to data analysis and machine learning. I have googled these topics and understand data leakage but it is not clear on when to perform outlier analysis.

To build a accurate and correct model, my understanding is:

  • Split dataset into train/test as first step and is done before any data cleaning and processing (e.g. null values, feature transformation, feature scaling). This is because the test data is used to simulate (see) how the model will perform if it was deployed in a real world scenario. Therefore you cannot clean/process the entire dataset.

  • Outlier detection (in general terms) should be done on the train dataset. This again simulates a real world scenario as the model will need to determine if there are any outliers and then take the correct action (e.g. remove, impute, cap to certain threshold). Checking outliers for the entire dataset (and doing some action) results in data leakage.

My qeustion is: should outlier detection/analysis be done on the training dataset or on the entire dataset before it is split into train/test?

I am trying to undersand what is the most common practice. I understand that outlier detection is not as straightforward as above as other factors may need to be considered.

Note: When searching CrossValidated, there are lots of answers regarding data leakage from train/test split, but there is no clear answer on when to remove outliers.

$\endgroup$
3
  • 1
    $\begingroup$ Why would you remove points? $\endgroup$
    – Dave
    Jul 6, 2021 at 13:59
  • 2
    $\begingroup$ Don't the definitions answer your questions? If you apply some preliminary procedure to the entire dataset, then no part of it legitimately qualifies for testing. $\endgroup$
    – whuber
    Jul 6, 2021 at 14:05
  • $\begingroup$ If available for your situation, using robust methods that limit the influence of outliers may make you free from the need to remove them or do anything specific on which later analysis then depends. Note however that this issue also affects the loss function used for cross-validation/evaluating test performance. This should be chosen in such away that it is not dominated by gross outliers. $\endgroup$ Jul 6, 2021 at 15:48

2 Answers 2

4
$\begingroup$

In my opinion you cannot remain vague about "outliers" when asking such questions. The answer to your question will most likely depend on what you mean by outlier and what procedure will be used to deal with outliers. A few imaginary scenarios:

  1. You have photographs of animals and some of them are damaged by technical errors. In this case you would simply discard them from the entire dataset as they would equally be discarded in, as you put it, real world scenario.

  2. You have gene expression data and some genes have abnormally high expression levels. You decide to deal with this by capping the expression at some arbitrary threshold $c$. Since this is a within-sample procedure - meaning the results will be the same regardless of whether you process each sample one by one or all of them together - you can again perform this before splitting into training and testing.

  3. You have similar gene expression data as before with some abnormally high values but you decide to do a cross-validation to get an optimal threshold parameter $c$. Now you actually would have to do such outlier "normalization" step not only separately for testing and training data, but separately for each cross-validation fold.

  4. You have customer data from an insurance company where samples can have missing features. You decide to impute those features using average values from the samples of the same class. Here you would have to perform this correction after splitting into training and testing. And again - if you do cross validation - separately in each cross-validation fold.

In summary, your general observation about checking whether this procedure would transfer to the "real world" setting is on point. Or alternatively - you could get intuition by pondering whether a certain procedure can be performed on a single sample (such procedures are called "in-sample" or "within-sample" procedures). As an example you cannot subtract a feature-wise mean from a single sample because you will get all 0s.

When dealing with an "out-sample" (between-sample) procedure you have to make sure that any estimation (a.k.a. "learning") is always done using only the data that is being used for estimation ("training data"). Then, once you get a value in this training data you have to use the obtained values on the testing data. And yes - simple things like centering the data by subtracting a feature-wise mean is also "learning". So you get the mean in the training step and subtract this training-data-obtained mean in the testing stage.

$\endgroup$
2
  • 1
    $\begingroup$ I really like point #1. The only change that I might make is to say that the criteria for determining if a photo is damaged should be developed on "in-sample" data. $\endgroup$
    – Dave
    Jul 6, 2021 at 15:32
  • $\begingroup$ @Dave thanks for the comment, feel free to edit as you see fit. I did made a short comment about "in-sample" procedures in point #2, but, called it "within-sample" (as opposed to "between-sample"). But maybe "in-sample" is a more proper term. $\endgroup$ Jul 6, 2021 at 15:36
1
$\begingroup$

If you're splitting the dataset to train/test or train/val/test, then you would "adjust the outliers" on the training set and then apply the change to test/validation set. Some good packages in python would be category_encoders or feature_engine.

$\endgroup$
1
  • $\begingroup$ It's unclear what "apply the change might mean," because identifying outliers is specific to the batch of data one is examining. How, then, does one translate an identification of outliers in a training set to some procedure on the testing set? Various possibilities come to mind, but they all depend on what you might mean by "outlier" and how you go about identifying one. These points are well made in the existing answer here. $\endgroup$
    – whuber
    May 17 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.