generic version of question: If you're comparing two statistical tests with different assumptions on the same data, and one gives a lower pvalue than another, does that mean that it has higher power?

biostatistics version: Comparing the SKAT and SKAT-O tests (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3440237/pdf/kxs014.pdf). SKAT-O is meant to have more power if the genetic variants being tested are unidirectional,so does that mean if it gives a lower (closer to zero) pvalue than SKAT that it does have more power in this scenario and I can assume that the genetic architecture is unidirectional and more like burden tests preferred setup? Im curious because knowing the answer to this would enable me to robustly decide which pvalues to use.


In general, the answer is NO. Suppose you have two different hypothesis tests $T$ and $T'$ for the same hypothesis test problem $H_0$ versus $H_1$ on the same data. Supposedly, $T$ and $T'$ uses different aspects of the data, for example, original data versus ranks. To make a meaningful comparison we must suppose that the two tests have the same significance level $\alpha$, (say =0.05). Or, at least, that is the usual approach.

But, often only p-values are reported without any prior choice of significance level, and the p-value is interpreted as some measure of "strength of evidence". If that is valid, a good measure of strength of evidence (Important: NOT strength of association or effect size!) is of course debated. If going that way, power is not a natural concept, since that depends on the (not choosen!) significance level. The idea, somehow, is that a p-value close to zero is strong evidence against the null hypothesis. That, at least, was Fisher's argument.

How can we now compare the hypothesis tests without the concept of power? We can look at the distribution of $P$ (the p-value). Under the null, for both tests, $P$ is uniformly distributed. We want a test that, under the alternative, tends to give small values of $P$. So now, the two tests can be compared on the basis of the distribution of $P$ under the alternative hypothesis. We want the test which give the $P$ that is "stochastically smaller" in some sense.

For (much more) about this approach, see the forthcoming https://www.bookdepository.com/Confidence-Likelihood-Probability-Tore-Schweder/9780521861601


The burden, SKAT, and SKAT-O tests represent 3 ways to pool information from low-frequency genetic variants so that relations of genomic loci to a biologic characteristic (phenotype) can be assessed. Burden tests assume that all low-frequency variants at a locus have the same relation to phenotype (unidirectional), so that variants all are pooled to obtain a single regression coefficient for the locus. The SKAT test instead treats variants as random effects, assuming a zero net effect among the variants and evaluating the magnitude of the variance of the phenotypic effects among genetic variants.

The SKAT-O is effectively a weighted combination of burden and SKAT tests, with the appropriate weight between burden (unidirectional) and SKAT (mean-zero) models determined from the data. It thus would be expected to perform better than burden tests or SKAT tests if there is a tendency toward one direction of phenotypic effect. In the linked paper describing SKAT-O, the authors did empirical power testing based on simulations and then examined a published data set with all these methods. They estimated the relative performance on the published data set by comparing p-values, presumably a part of the basis for this question.

In the context of that paper, that use of p-values to evaluate some closely related tests on the same data set makes sense. In general, however, general statements about relations of p-values to power can be misleading, as @kjetil b halvorsen notes in another answer here.

If you are considering analysis of your own data with these methods, consider your knowledge of the genomic loci first. Do not run all 3 tests and simply choose the one that provides the lowest p-value. If you don't have prior knowledge about the nature or effects of genomic variants at your loci of interest, the SKAT-O test would seem to be preferable as it will choose the best weight between the burden and SKAT models from your data. That will use up one extra degree of freedom (maybe 2) for statistical tests, but with a large number of variants that should not make much practical difference in terms of power.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.