# Generate null distribution from pvalues

I have a set of experiments on which I apply the Fisher's exact test to statistically infer changes in cellular populations. Some of the data are dummy experiments that model our control experiments which describe the null model (Type Column).

This is what the data looks like (link):

I am applying the Fisher's exact test on populations under 2 conditions (UNTREATED, TREATED) under the same ID.

data.merged = merge(x=subset(df,condition == 'UNTREATED'), y= subset(df, condition =='TREATED')), by ='ID')

However, due to some experimental variation most of the controlled experiments reject the null hypothesis at a $$p_{val} <0.05$$. Some of the null hypotheses of the actual experimental conditions are also rejected at a $$p_{val} <0.05$$. However, these pvalues, are magnitudes lower than those of my control conditions. This indicates a stronger effect of these experimental conditions. However, I am not aware of a proper method to quantify these changes and statistically infer them.

An example of what the data looks like:

ID      Pval            Condition
B0_W1   2.890032e-16    DUMMY
B0_W10  7.969311e-38    DUMMY
B0_W11  8.078795e-25    DUMMY
B0_W2   3.149525e-30    Gene_A
B1_W1   3.767914e-287   Gene_B
B1_W10  3.489684e-56    Gene_X
B1_W10  3.489684e-56    Gene_Y

1. selecting the ctrl conditions and let $$X = -ln(p_{val})$$ which will distribute the transformed data as an expontential distribution.
2. Use MLE to find the $$\lambda$$ parameter of the expontential distribution. This will be my null distribution.
3. Apply the same transformation to the rest of the $$p_{val}$$ that correspond to the test conditions
4. Use the cdf of the null distribution to get the new "adjusted pvalues".

This essentially will give a new $$\alpha$$ threshold for the original pvalues and transform the results accordingly using the null's distribution cdf. Are these steps correct? Is using MLE to find the rate correct or it violates some of the assumptions to achieve my end goal? Any other approaches I could try?

• I'm having a little trouble understanding the question. It kind of sounds like you want to use the p-value to measure the effect size of test vs control. Why not fit a model with a parameter for treatment effect?
– Eli
Commented Oct 8, 2020 at 17:57
• @Eli I am not sure I understand what you are suggesting? Fitting in a generalised linear model the count data instead of fisher's exact test? What I am having difficulty using this method is interpreting the results I am going to get from such a glm. Can I do statistical inference? Excuse my ignorance, I just find very confusing the fact that my CTRL experiments are found to be significant. Yes, that is accurate I am trying to measure the effect size compared to the control using p-values. Commented Oct 9, 2020 at 16:31
• That is what I was suggesting. Can you give an example of your dataset before performing any type of test or model? It looks like the data you provided is what you get after fitting an Exact Test. You can change the values so you're not sharing your real data.
– Eli
Commented Oct 9, 2020 at 16:53
• @Eli I edited the question and added the information with a paste bin URL. Commented Oct 12, 2020 at 10:01
• @Eli in light of your suggestion my question is really is: Is glm a better tool than Fisher's exact test of infering the changes in the population? Other than than, what I thought on p-values it does not look right to me but it does achieve what I want which is the normalization of the p-values which is what I wanted in the first place. I found no material which has any methods of normalizing pvalues on noisy data. I find information only on multiple testing correction. Commented Oct 13, 2020 at 11:23

As noted in the comments, it would probably be best to approach this using a multilevel logistic regression model, along the lines of

glmer(cbind(Population_Pos, Population_Neg) ~ Condition * Type + (1|ID)

but since that's not your question, I won't go into depth on this.

On the approach your have been taking, I don't think your idea with transformed p-values works, but there is a simpler solution. This assumes you have a large number of control experiments.

1. For each experiment, calculate a measure of the effect size. This could be the Chi-squared statistic (calculated for the Fisher exact test), but it probably makes more sense to use the odds-ratio: $$\frac{\text{Pos}_{\text{Treated}}}{\text{Neg}_{\text{Treated}}} \div \frac{\text{Pos}_{\text{Untreated}}}{\text{Neg}_{\text{Untreated}}}$$.
2. Plot the distribution of effect sizes for the active experiments and the control experiments. You should see larger effects for active experiments.
3. For any active experiment, you can check how many control experiments had a larger effect size. For example, if an active experiment has an effect size greater than that in 99% of control experiments, the one-tailed p-value for the null hypothesis that the effect size for this experiment comes from the same distribution as the control effect sizes is $$p < .01$$.

Again, though, this is an approximation that only works when you have a very large number of control experiments. If you do in fact have only 8 control experiments, this approximation isn't very useful at all!

--

To elaborate on why this is different to what you propose, first note that a p-value is not a measure of effect size, and depend on the sample size, so you're not answering the same question if you use the p-values here. Second, we can't really assume a distribution for the p-values (or effect sizes) from the control experiments, since it seems that the null hypothesis isn't true for them (if the null was true, p-values would be uniformally distribution between 0 and 1). That's why I suggest this non-parametric approximation instead.