1
$\begingroup$

Suppose I have a continuous random variable which is bound between 0 and 1. The distribution is left skewed like the picture below: enter image description here

My goal is to identify outliers that are small or farther away from 1. In other words, I don't care about outliers that exist on the right side.

In an attempt to perform outlier detection, I decided that I could represent the data with a Beta distribution and use scipy to estimate the parameters of that distribution. Then, I computed the CDF and then chose a probability of observing a value as great as X_c, P(X_c>x), such that every point, x, below X_c is anomalous.

Does this approach seem reasonable? I could not find any resources on using a beta distribution to model data unless it was being used as a prior for bayesian updating. The distribution seems appropriate to me for my use case due to its ability to model the shape of the data I am dealing with and because it is a valid probability distribution.

$\endgroup$
  • 1
    $\begingroup$ This is a good way to identify observations with small values, does not make them outliers though. What are you trying to achieve here? $\endgroup$ – user2974951 Aug 21 '19 at 7:36
  • 1
    $\begingroup$ Definitely anything smaller than 0 :) $\endgroup$ – wolfies Aug 21 '19 at 7:48
  • 1
    $\begingroup$ What you proposed could work, especially if you have some sort of bimodal distribution, for ex. some CPU's just fail and are much worse then the others on the lower end. However, if this is not true, observations with small values could just be random, as expected by the distribution, and so you would be wasting your time analyzing them. $\endgroup$ – user2974951 Aug 21 '19 at 8:02
  • 2
    $\begingroup$ The beta distribution is used (e.g.) to model cloudiness in meteorology. But using a beta distribution as a reference raises as many questions as it answers. Why not just use a quantile plot? A histogram is at best an indirect way to look at detail in the lower tail. $\endgroup$ – Nick Cox Aug 21 '19 at 8:16
  • 1
    $\begingroup$ I like the idea, but wish to point out that adopting a Beta assumption may be a bit too strong. In effect, you are postulating that the left tail eventually decays like a power of $x.$ That (much weaker) assumption may be all you need to construct an effective outlier-flagging procedure. The foregoing objections (possible bimodality, etc) are not problems when you're using your procedure to screen data for further evaluation. From this perspective, it is important only that (1) you not classify too many good observations as outlying and (2) you find a large proportion of the bad data. $\endgroup$ – whuber Aug 21 '19 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.