I would like to have FDR/estimated FDR from two distributions with different mean and similar sigma. One of this is a reference distribution of 'real' samples (TP) while the other one is predicted (containing FP + TP).

Of course I could generate a null distribution and then just estimate FDR with that. I was wondering whether having the knowledge what is TP and what is FP will allow to do something as naive as just taking the predicted samples calculate local FDR directly (TP/P) and then filter for whatever threshold.

  • $\begingroup$ The trick that may work under some conditions (good separation between your models and large sample size): you take all the values from FP+TP and take their p-values according to the distribution of TP. 2 * (amount of p-values > 0.5) will give you an expected number of TPs in your FP+TP. In general your task is "estimate amount of FP in FP+TP distribution" - it is the job for mixture model analysis. $\endgroup$ Oct 5 '19 at 8:21
  • $\begingroup$ The two mean of the two distributions are very different (p < 10-68). So the conversion to p will be the test of mean difference with permutation test / bootstrap? Also for do you mean gaussian mixture model with 2 components? $\endgroup$
    – D.A.
    Oct 6 '19 at 10:29
  • $\begingroup$ No no, p value is not from the test, p value is from the distribution of true positives. Basically, how big is the proportion of true positives bigger than the observed value. And yes, gaussian mixture will be fine, parameters for one model you can estimate from your tp distribution. $\endgroup$ Oct 6 '19 at 14:49
  • $\begingroup$ Okok, so just quantile right? Also, which assumptions are we making? normality of both distribution? Sorry I am not the best statistician in the world $\endgroup$
    – D.A.
    Oct 7 '19 at 8:38
  • $\begingroup$ yeap. if the distributions intersect largely - go from 50% quantile to like 10% or 20% top quantile from TP distributions, but then you will need to multiply by 10 or 5 (since p-value is uniformly distributed for the large enough sample). Actually there is no assumptions. If you do mixture analysis - you assume some family of distributions, but here I'd say no. $\endgroup$ Oct 7 '19 at 8:48

To estimate FDR you need to estimate the number of True Positives in your mixture.enter image description here (I took this picture from here).

Here the example is mixture of normals, but in theory the approach should work for any distribution (reasonable ones with reasonable sample sizes and intersections). So, what do you have:

1) green density of TP+FP 2) orange density of TPs

The key observation: how many FPs are on the right from the top-10% quantile of orange distribution (the exact value on x-axis may be around 6 in the particular picture)? Almost 0. And you know that 10% of TPs are on the right of this threshold.

Basically, you find how many of your values that form the green density are on the right of this threshold. Then you multiply this number by 10 (since we know that 10% of TPs are on the right). This is how you estimate the number of TPs in the mixture of TPs and FPs.

Of course, these values (10% or 5% or 50%) should be adapted depending on how big is the intersection between the components. If there is only 1% difference - well, then you'll need a lot of samples and 1% threshold.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.