I want to test if a set of nodes have more or less links to another group than what it should be at random. Since I should not do any biased estimation of the direction of my test, I want to apply a two-sided test.

The idea extracted from literature is to calculate both one tailed p values, take the minimum and multiply it by two. However, some of my null distributions looked highly right skewed and have few different values (all are positive), see example:

enter image description here

Checking this question P-value in a two-tail test with asymmetric null distribution and regarding @Scortchi answer (which I guess it was extracted from "Concepts of Nonparametric Theory" by J.W. Pratt and J.D. Gibbons.), I have tried his suggestion since some of my null distributions are not symmetric.

"Adding to the smaller one-tailed p-value the largest achievable p-value in the other tail that does not exceed it."

However, with the kind of null distribution showed above (low variability and near 0), there are huge differences between possible p values. Since I am just using the following method to calculate the p value:

$$ p =\dfrac{(w+1)}{(N+1)}, $$

Where $w$ is the number of values in the distribution at least as extreme as the observed value. Therefore, having very few different values in the distribution, it generates very discrete p values.

Applying what was suggested for asymmetric discrete distributions would lead to a less conservative p value, than just doubling the minimum one tailed p value (upper bound).

Checking both p value distributions when testing False Positives we have:

1) p values calculated as the minimum of both one tailed p values multiplied by two

2) p values calculated by adding to the smaller one-tailed p-value the largest achievable p value in the other tail that does not exceed it.

enter image description here

As observed, p value distribution is uniformly distributed for the first case , but in the second case, p values close to 1 decrease.

Overall results are very very similar, so the question is:

Is it really a big deal, using one or another? Could for instance, doubling the minimum one tailed p value consider a wrong approach, even if it is an upper bound result?

  • $\begingroup$ I think the null distributions means the distribution of test statistics when null hypothesis is true. For example, the statistics for t-test follows t distribution if null hypothesis is true. What is your test statistics and what distribution it follows when null hypothesis is true? $\endgroup$
    – user158565
    Oct 10, 2018 at 2:38
  • $\begingroup$ This quandary is removed (or at least moved somewhere else, where it may be easier to deal with) if you first decide on a two-tailed rejection rule for your hypothesis test, since the corresponding two tailed p-value is then defined (the lowest significance level that would still lead to rejection). The issue of defining the two-sided rejection region remains, but the levers (and implied tradeoffs) involved in that decision are more clear (IMO); you choose the properties of your tests / which samples will lead to rejection and the rest follows from there. $\endgroup$
    – Glen_b
    Oct 10, 2018 at 5:01
  • $\begingroup$ @a_statistician The distribution of test statistics is built based on sampling from a network, emulating the properties of the original group $A$ (size and degree of the nodes), the idea is to know the distribution of links $k$ that would be expected from random groups of the same properties of the original group A to a second group $B$. Once saying this, the distribution is normally distributed however problems arises when the values of the links are really low (as stated in my question, in the first figure). When this happens the distribution is asymmetric and right skewed. $\endgroup$
    – user219773
    Oct 10, 2018 at 8:11
  • $\begingroup$ @a_statistician The p values are calculated in a non-parametric way, just based on how many values are greater of equal my real value (from all the random samplings that I do, lets say 2000, see how many of those groups give more extreme number of links than my real group). $\endgroup$
    – user219773
    Oct 10, 2018 at 8:12
  • $\begingroup$ @Glen_b My test statistics resembles the permutation test in the way that I generate the distribution under the test statistics by sampling from the network (taking nodes with similar properties in size and degree of the nodes as the original group). The distribution will end up being the number of links you are expecting from these random groups. Then the p-value will be just the proportion of values from that distribution that are greater or lower than the real value. Once stating this, my rejection rule will be basically the quantiles 1−α/2 and α/2, since I am applying a two-tailed test. $\endgroup$
    – user219773
    Oct 10, 2018 at 16:29


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.