3
$\begingroup$

I am using pyod to detect outliers in data, and I came across this official example: https://github.com/yzhao062/pyod/blob/master/examples/comb_example.py

I have a question regarding the need to split the data into X_train, X_test, y_train, y_test when detecting outliers. What is the purpose of this data split? I thought outlier detection was typically performed in an unsupervised setup, so I'm wondering why such a split is necessary.

$\endgroup$
4
  • 1
    $\begingroup$ Thank you for showing me this great piece of software! During the training of anomaly detection algorithms some parameters of the model are adjusted to learn what is normal. To test how many anomalies the algorithm reports on unseen test data, you need to hold these samples out. $\endgroup$
    – Ggjj11
    Commented Jun 26, 2023 at 18:23
  • 1
    $\begingroup$ Why was my answer converted to a comment? The purpose of a holdout test data set is just to computer performance metrics. $\endgroup$
    – Ggjj11
    Commented Jun 27, 2023 at 5:27
  • $\begingroup$ Is this method used as part of exploratory data analysis to find potential outliers, or are you building a model to detect outliers? $\endgroup$
    – Eli
    Commented Jun 29, 2023 at 15:26
  • $\begingroup$ @Eli this should be part of the answer to provide clarity on when to use a split and when not to. It should be clear at which stage of the process you need a split and when you do not. $\endgroup$
    – JAdel
    Commented Jun 29, 2023 at 15:52

1 Answer 1

1
$\begingroup$

As you can see in the paper to which the repository belongs to, there is a distinction made between unsupervised and supervised anomaly detection. enter image description here

In this repository this is done, as it part of a outlier detection benchmark study. So here another reason for using train and test splits is to compare different outlier detection methods on robustness and other things. So it makes a sense use a train and test set to evaluate, whether a outlier detection strategy might misclassify samples, due to biases in the training data.

In the paper they also give additional explanations for specific scenarios why the split data. enter image description here

Usually, when we think about outlier detection we think about the univariate case, where we only look at a single variable to identify an outlier. But it can be more complicated in multivariate case, where each individual variable is not an outlier but the combination of these values is unlikely. It makes sense for example to use a train and test set when, you want to identify outliers based on your x and y(target variable) values. So an outlier could be a case, where both the input variables are not usual as well as the target variable. But this combination of both might be unusual.

A second argument for splitting train and test set for outlier removal it to ensure no data leakage. Like we split the data into train and test set before basing out normalisation on the train set, one can also argue also to build your outlier rules only based on the train set.

So in my opinion if you are building machine learning models, you should base your outlier detections on the train set, and apply the same rules to the test set. So for example determine the Interquartile range based on the Train set and use the exact values for the testset.

If you are interested more in inferential statistics, then I would not use a train and test set split. I mean you usually dont do it anyways. Because here you are not interested in the best performance, rather in informative parameters. So data leakage and a resulting overestimation of model performance is not of great concern.

So more generally, when you split data anyways then make use of the split for outlier detection as well.

$\endgroup$
1
  • $\begingroup$ Hi Janosch. Thank you for your answer! Actually, I am looking for a canonical answer when to use a train and test split, especially in which context it is necessary or not. Could you make your answer more canonical and provide an example for a case when to use a split and when not to? (In context of outlier detection algorithms) $\endgroup$
    – JAdel
    Commented Jul 3, 2023 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.