1
$\begingroup$

One method to detect outliers in a dataset $[x_1 ... x_N], x_i \in R$ consists in finding the samples $x_i$ such that $$ x_i \lt Q_1-K*IQR | x_i \gt Q3 + K*IQR $$ where $Q_1$ and $Q_3$ are the first and third quartile, respectively, $IQR$ is the interquartile range and $K$ is a constant (e.g. 1.5). How can this method be applied to multivariate dataset $[x_1 ... x_N], x_i \in R^k$ , with a high number of dimensions (e.g. k > 100)?

$\endgroup$

1 Answer 1

1
$\begingroup$

Detecting outliers can be done through univariate or multivariate frameworks. I recommend, detecting extreme outliers (also called global filtering) is better to be done via IQR with a univariate approach while detecting local/spatial outliers is better to be done with a multivariate/multidimensional fashion through density-based algorithms.

Edited:

Adding another point to complete the answer! The choice of the algorithm is highly dependent on the sample size and heterogeneity characteristic of your data!

$\endgroup$
6
  • 1
    $\begingroup$ What is the principle behind deletion of outliers? Why risk having your results non-prospectively-applicable, i.e., not applying to future data that may contain "outliers"? $\endgroup$ Dec 9, 2021 at 13:12
  • $\begingroup$ Outliers leads to the violation of underlying structure in your data. Hence results into biased decisions. TO your second question, this is absolutely depend on your application. If you need a streaming outlier detection algorithm, then it is recommended to go with machine learning approaches which are regression outlier based not Hawkins outlier. $\endgroup$
    – morteza
    Dec 9, 2021 at 15:55
  • 2
    $\begingroup$ But any definition of "outlier" is arbitrary and will limit generalizability. Better to use robust statistical methods not hurt by "outliers". $\endgroup$ Dec 9, 2021 at 16:20
  • 1
    $\begingroup$ Good point. Can you name some cool robust statistical methods for finding local outliers? $\endgroup$
    – morteza
    Dec 9, 2021 at 17:05
  • 1
    $\begingroup$ No, I'm not referring to robust outlier detection. I was referring to robust estimators / model fitting methods so that results are meaningful whether "outliers" are present or not. $\endgroup$ Dec 9, 2021 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.