2
$\begingroup$

How can we find outliers in a dataset with a (highly) skewed distribution? With a normal distribution, is it well documented to use 2 x Standard Deviation or the upper boundary of the box plot (1.5 x IQR). However, for something like a conversion rate, where the distribution is severely positively skewed, how can we find the high conversion rates that are outliers?

I have done a lot of research around this topic and is a big business problem for us, but I can't find much at all about it.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ If your distribution is known to be normal, then points beyond 2 SD from the mean will be common, but there is no need for identifying outliers any way. The criterion that points more than 1.5 IQR from the nearer quartile (below the lower quartile as well as above the upper quartile) was never intended by J.W. Tukey, who suggested it, as a hard criterion for outliers. It is just a criterion for plotting points separately before thinking about them. $\endgroup$
    – Nick Cox
    Commented Feb 17, 2020 at 11:54
  • $\begingroup$ Your tagging of outliers underlines that there are many threads here on that topic, some much upvoted. I can't see that you have a distinctively new question here. WIth highly skewed distributions, either work on a transformed scale, or consider what skewed distribution might make sense of the data. Real data often include the Amazon, or Amazon, genuinely very big values. $\endgroup$
    – Nick Cox
    Commented Feb 17, 2020 at 11:56
  • 1
    $\begingroup$ Maybe duplicate: stats.stackexchange.com/q/129274/103153 $\endgroup$
    – Lerner Zhang
    Commented Feb 17, 2020 at 12:13
  • 1
    $\begingroup$ Does this answer your question? Outlier Detection on skewed Distributions $\endgroup$
    – Lerner Zhang
    Commented Feb 17, 2020 at 12:14

1 Answer 1

Reset to default
0
$\begingroup$

To define or filter out outlier, first we need to define what is "normal scenarios".

We do not need to assume data is coming from normal distribution. If we are using parametric method, any distribution should be fine to get the outliers. For example, we can fit our data to an exponential distribution. After fitting, we can calculate the likelihood for any data point that belong to this distribution and use likelihood to detect outliers.

Here is an example:

  • We first generate data with rate $1$ from exponential distribution.

  • Then we fit the a model on data and got rate $0.97$ (pretty close to $1$ with 1000 samples).

  • Finally we can test for different points: $1, 3, 30, -1$. From the numbers we can see, 30 and -1 are outliers (PDF values close to 0).

 

> require(MASS)
> set.seed(0)
> x=rexp(1000)
> hist(x)
> fit1 <- fitdistr(x, "exponential") 
> print(fit1$estimate)
rate 
0.9711787 
> dexp(1,rate=fit1$estimate)
[1] 0.3677237
> dexp(3,rate=fit1$estimate)
[1] 0.05271892
> dexp(30,rate=fit1$estimate)
[1] 2.157607e-13
> dexp(-1,rate=fit1$estimate)
[1] 0
Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.