I have:

  • About fifty values $x_1,…,x_n$ sampled from the same but unknown distribution $X$. I have no useful theoretical insight on the nature of that distribution. Empirically, the distribution is considerably prone to outliers (heavy tail).
  • A single value $y$ for which I would like to estimate the probability $p$ that it was sampled from the above distribution.
  • Several iterations of the same scenario, but with the distribution of $X$ varying (including its general form). Using a subset of the data with a known ground truth, these allow for a benchmark my choices (the details of which go beyond the scope of this question).

Since I have several iterations, any solution must work automatically. For example, I cannot prune outliers by hand (not that I would consider this a particularly good approach).

What I considered so far

I know that my problem boils down to estimating the CDF of $X$. I can then obtain a one-sided $p=\text{CDF}(y)$ and from this it’s straightforward to get a two-sided $p$. I see three general ways to do that:

  1. Just rank all values with $r_y$ being the rank of $y$. Then I can compute: $\text{CDF}(y) = \frac{r_y+1}{n+1}$ (for low $y$).

  2. Create a kernel-density estimator $K$ and use its CDF. (I have means to obtain a reasonable estimate of the kernel width.)

  3. Assume normality (or some other parametric distribution), fit its parameters and use the resulting CDF.

Now, a rank-based approach (1) is not overly sensitive to outliers, however it’s too radical in not making a distinction between close and extreme outliers. The kernel-based approach (2) is overly sensitive to outliers (at least for a Gaussian kernel and any reasonable kernel width). Using some parametric distribution (3) is far out and cannot handle outliers at all. My benchmark results support this.

Now, the best way forward for me seemed to just mix the ranked- and kernel-based approach, i.e., average their CDFs (possibly with a weight). My benchmark confirms this and yields a higher performance than for the unmixed approaches. This is reasonably robust with respect to the mixing ratio: Anything between roughly 3:7 and 7:3 (kernel to ranked) yields the same good result.


My mixing approach feels awkwardly arbitrary, although I have a benchmark that at least somewhat supports it.

  • Is there any reasonable argument or reference for the mixing approach?
  • Is there any approach that is similar in nature (and might perform even better)?
  • Is there another relevant approach that I am missing?
  • 2
    $\begingroup$ The kernel approach is arbitrary because its results are (exquisitely) sensitive to what is ordinarily considered an incidental property of the kernel: namely, the heaviness of its tails. Its results are also quite sensitive to the kernel width, which--if you are using standard methods to estimate it--has not been based on your problem, but is based on generating a "nice" estimate for the interior of the distribution and not the tails. A suitable parametric distribution can handle outliers well, but the results are sensitive to the family you use. $\endgroup$
    – whuber
    Jun 30 at 16:19
  • 2
    $\begingroup$ (Continued) Some of the most effective approaches are derived from theoretical considerations of what the tail of the distribution ought to look like. You don't write anything about that, but if anything is known, I would encourage you to think in that direction. $\endgroup$
    – whuber
    Jun 30 at 16:20
  • 2
    $\begingroup$ Note that even if you know the distribution of $X$ you can't calculate the "probability that $y$ was sampled from the same distribution". What you can do is calculate a $p$-value that might allow you to reject that hypothesis - those are different things. $\endgroup$
    – J. Delaney
    Jun 30 at 17:18
  • 1
    $\begingroup$ @whuber: At best, I might empirically estimate the nature of the tail of the distribution. I’ll have to think about that. $\endgroup$
    – Wrzlprmft
    Jun 30 at 22:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.