Generate null distribution from pvalues I have a set of experiments on which I apply the Fisher's exact test to statistically infer changes in cellular populations.
Some of the data are dummy experiments that model our control experiments which describe the null model (Type Column).
This is what the data looks like (link):

I am applying the Fisher's exact test on populations under 2 conditions (UNTREATED, TREATED) under the same ID.
data.merged = merge(x=subset(df,condition == 'UNTREATED'), y= subset(df, condition =='TREATED')), by ='ID')

However, due to some experimental variation most of the controlled experiments reject the null hypothesis at a $ p_{val} <0.05$. Some of the null hypotheses of the actual experimental conditions are also rejected at a $ p_{val} <0.05 $. However, these pvalues, are magnitudes lower than those of my control conditions. This indicates a stronger effect of these experimental conditions. However, I am not aware of a proper method to quantify these changes and statistically infer them.
An example of what the data looks like:
ID      Pval            Condition
B0_W1   2.890032e-16    DUMMY 
B0_W10  7.969311e-38    DUMMY
B0_W11  8.078795e-25    DUMMY   
B0_W2   3.149525e-30    Gene_A
B1_W1   3.767914e-287   Gene_B
B1_W10  3.489684e-56    Gene_X
B1_W10  3.489684e-56    Gene_Y

One idea I had:

*

*selecting the ctrl conditions and let $  X = -ln(p_{val}) $ which will distribute the transformed data as an expontential distribution.

*Use MLE to find the $\lambda$ parameter of the expontential distribution. This will be my null distribution.

*Apply the same transformation to the rest of the $p_{val}$ that correspond to the test conditions

*Use the cdf of the null distribution to get the new "adjusted pvalues".

This essentially will give a new $\alpha$ threshold for the original pvalues and transform the results accordingly using the null's distribution cdf. Are these steps correct? Is using MLE to find the rate correct or it violates some of the assumptions to achieve my end goal? Any other approaches I could try?
 A: As noted in the comments, it would probably be best to approach this using a multilevel logistic regression model, along the lines of
glmer(cbind(Population_Pos, Population_Neg) ~ Condition * Type + (1|ID)
but since that's not your question, I won't go into depth on this.

On the approach your have been taking,
I don't think your idea with transformed p-values works,
but there is a simpler solution.
This assumes you have a large number of control experiments.

*

*For each experiment, calculate a measure of the effect size. This could be the Chi-squared statistic (calculated for the Fisher exact test), but it probably makes more sense to use the odds-ratio:
$\frac{\text{Pos}_{\text{Treated}}}{\text{Neg}_{\text{Treated}}} 
\div 
\frac{\text{Pos}_{\text{Untreated}}}{\text{Neg}_{\text{Untreated}}}
$.

*Plot the distribution of effect sizes for the active experiments and the control experiments. You should see larger effects for active experiments.

*For any active experiment, you can check how many control experiments had a larger effect size. For example, if an active experiment has an effect size greater than that in 99% of control experiments, the one-tailed p-value for the null hypothesis that the effect size for this experiment comes from the same distribution as the control effect sizes is $p < .01$.

Again, though, this is an approximation that only works when you have a very large number of control experiments. If you do in fact have only 8 control experiments, this approximation isn't very useful at all!
--
To elaborate on why this is different to what you propose,
first note that a p-value is not a measure of effect size,
and depend on the sample size, so you're not answering the same question if you use the p-values here.
Second, we can't really assume a distribution for the p-values (or effect sizes) from the control experiments, since it seems that the null hypothesis isn't true for them (if the null was true, p-values would be uniformally distribution between 0 and 1). That's why I suggest this non-parametric approximation instead.
