I have a data set that consists of many single data points. They are the measurements of network traffic, so they include e.g. '1403021', '1402341, '1399312'... values that are labeled as 'label1' and e.g. '1031', '301', '501', '10'... values that are labeled as 'label2'. All the data can be measured in date range, so the data set could consist a one week sampling. So my point is that I have a lot of data that is not related to each other but I can calculate 'something' for each label. My question is how can I accurately say that some single data points in data set of e.g. 'label2', are 'high rise' (1031) points or 'low rise' (10) points - so points that by eye tells us that something odd is happening in this measurement?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Are you interested in outliers in the univariate data set of label2, or in the bivariate dataset (label1, label2)? $\endgroup$– cdalitzCommented Sep 29, 2021 at 8:26
-
$\begingroup$ I am interested in both labels but separately. There are also other labels that have a different set of values but I need to see outliers in all indexes separately. $\endgroup$– norivotsetCommented Sep 29, 2021 at 8:34
-
$\begingroup$ As you are new contributor, here is a suggestion: if an answer actually resolves your question, mark it as "accepted". This will remove your question from the unanswerd questions list. $\endgroup$– cdalitzCommented Oct 2, 2021 at 6:02
-
$\begingroup$ It didn't resolve my question. $\endgroup$– norivotsetCommented Oct 3, 2021 at 8:11
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If you are only interested in outliers in one variable $X$, you can simply estimate the probability density $f(x)$ of this variable from the data and define outlieres as values $x$ with $$ \alpha/2 > P(X<x) = \int_{-\infty}^x f(x')\, dx'$$ For large values the citerion is anlogous, i.e.: $P(X>x)<\alpha/2$.
Estimation of $f(x)$ can be done either parametric (e.g. a normal density) or non-prametric (kernel density estimator), as provided by the R function density(). In the latter case, the integration must be done numerically, for example with Simpson's rule (this has the advantage of working with equidistant function sample points).
If the distribution is multimodal and outliers can lie in the middle between maxima, it might be the simplest to base outlier detection on distance statistics. See section 5 "Distance rejection" in this report.
