I want to test if a set of nodes have more or less links to another group than what it should be at random. Since I should not do any biased estimation of the direction of my test, I want to apply a two-sided test.
The idea extracted from literature is to calculate both one tailed p values, take the minimum and multiply it by two. However, some of my null distributions looked highly right skewed and have few different values (all are positive), see example:
Checking this question P-value in a two-tail test with asymmetric null distribution and regarding @Scortchi answer (which I guess it was extracted from "Concepts of Nonparametric Theory" by J.W. Pratt and J.D. Gibbons.), I have tried his suggestion since some of my null distributions are not symmetric.
"Adding to the smaller one-tailed p-value the largest achievable p-value in the other tail that does not exceed it."
However, with the kind of null distribution showed above (low variability and near 0), there are huge differences between possible p values. Since I am just using the following method to calculate the p value:
$$ p =\dfrac{(w+1)}{(N+1)}, $$
Where $w$ is the number of values in the distribution at least as extreme as the observed value. Therefore, having very few different values in the distribution, it generates very discrete p values.
Applying what was suggested for asymmetric discrete distributions would lead to a less conservative p value, than just doubling the minimum one tailed p value (upper bound).
Checking both p value distributions when testing False Positives we have:
1) p values calculated as the minimum of both one tailed p values multiplied by two
2) p values calculated by adding to the smaller one-tailed p-value the largest achievable p value in the other tail that does not exceed it.
As observed, p value distribution is uniformly distributed for the first case , but in the second case, p values close to 1 decrease.
Overall results are very very similar, so the question is:
Is it really a big deal, using one or another? Could for instance, doubling the minimum one tailed p value consider a wrong approach, even if it is an upper bound result?
