8
$\begingroup$

I identified SNPs that are associated with phenotype through GWAS. I labeled the types of genetic variants in those significantly associated SNPs and now I'm trying to assess the association of those variants with the phenotype with Fisher's exact test.

Looking at the output of the Fisher exact test in R, can one just by looking at the odds ratio and the p-value state that the genetic variant of interest is either enriched or depleted?
Having for example a p-value = 0.0003 and an OR=0.24 as the result in testing the association between ncRNA and phenotype, how would one interpret the OR? Can we interpret this in the context of gene expression as well?
I've read somewhere that for a 2x2 contingency table, the formula for computing the odds ratio and the fold change in expression analyses are the same thing. Please correct me if I'm wrong.
Is there a connection between Fisher's exact test and differential gene expression analysis?

Improve this question
$\endgroup$
4
  • $\begingroup$ to use this network effectively, do not edit the question so that it is no longer useful to anyone but yourself. The question should be a question. If you have the answer now, post it as an answer and allow the community to vote on it (as compared to Prof. Harrell's excellent take). $\endgroup$
    – AdamO
    Commented Feb 15 at 16:07
  • $\begingroup$ Link-only answers should, at best, be a comment. Better yet, take the time to spell out the ideas in the link you provided here: academic.oup.com/bioinformatics/article/23/4/401/181853 $\endgroup$
    – AdamO
    Commented Feb 15 at 16:08
  • $\begingroup$ When you say you "identified SNPS... through GWAS" what exactly did you do? There are virtually thousands of methods which could be called GWAS. The real questions are 1. Did you control FDR or FWER? (If FDR, making individual comparisons is at best tenuous, borderline deceptive). And 2. If you use an association measure and significance test different from that of the workhorse behind the GWAS method, why would you expect them to agree? $\endgroup$
    – AdamO
    Commented Feb 15 at 16:13
  • $\begingroup$ @AdamO 1. The FDR was controlled at 0.1 significance. Why is making individual comparisons borderline deceptive? 2. My goal with the Fisher exact test was not to determine significance, but enrichment. Therefore, with my question I did not reach to this significance test to include an estranged significance in addition to the GWAS results, but to understand the interpretation of a small OR and a significant p-value which in my case of analysis indicates significant depletion of ncRNA. I was not aware at the time, that many software for enrichment analysis are based on Fisher exact. $\endgroup$
    – mango
    Commented Feb 16 at 8:51

1 Answer 1

Reset to default
10
$\begingroup$

As has been thoroughly discussed on this site, Fisher’s “exact” test is not very accurate, the ordinary Pearson $\chi^2$ is more accurate, and you can save a lot of computing time by using it.

A more important challenge underlying your question, no matter which associate measure you use to describe the results, is that it’s too easy to commit an “absence of evidence is not evidence of absence” error, i.e., in concluding that $p > 0.05$ means no association. A simple Bayesian analysis would expose the problem. Take a reasonable skeptical prior distribution and analyze the candidate features one-at-a-time. For each one compute the posterior probability that the true unknown odds ratio exceeds 1.2. You’ll see a minority of features for which you can learn something, i.e, that the posterior probabilities are > 0.95 of < 0.05. Now compute the probabilities that the odds ratios are close to 1.0, e.g, between 4/5 and 5/4, indicating that there is little association. You’ll see the vast majority of features having probabilities of minimal effect in the range of [0.2, 0.8], i.e., we can’t say one way or another where the feature is associated with some outcome. This exposes the meaninglessness of significance testing in this context.

Look here for simple simulations showing that below a certain sample size there is no correlation between the true odds ratios and estimated odds ratios. Scary, but shows that you need high sample sizes to learn in high dimensions when the learning is not biologically driven. Examples in the link also show how to use the bootstrap to expose the difficulty of the task through confidence intervals on importance measures or their ranks.

Improve this answer
$\endgroup$
6
  • 4
    $\begingroup$ Hi Frank. Can you indicate the sense in which you mean inaccurate in relation to the Fisher exact test and the circumstances in which it is less so than the Pearson chi squared? In very large samples they tend to behave similarly, and in very small samples at least in the 2x2 case its very difficult to get their exact distributions to differ by much at all (when conditioning on the margins for both). Are you referring to the lack of available significance levels in very small samples when conditioning on the margins (which impacts both significance level and power)? $\endgroup$
    – Glen_b
    Commented Jan 22 at 17:12
  • 1
    $\begingroup$ Please see hbiostat.org/bbr/prop.html#fishers-exact-test . Pearson was wrong when he guessed that you need an expected frequency > 5 for his test to be accurate. It’s more like 1.0. $\endgroup$
    – Frank Harrell
    Commented Jan 23 at 12:31
  • $\begingroup$ Thanks. I know the Expected > 5 rule is not correct; in some circumstances much smaller values lead to acceptable approximations; though in other circumstances 5 is not really sufficient. I'm not sure I agree with the idea that a conservative test correspond to p-values being too large; there's two different issues that seem to be being conflated. I agree that it's sometimes quite conservative because there's a lack of available significance levels, so sometimes alpha is considerably smaller than desired. The claim that p-values are too large is a separate issue, involving a different argument $\endgroup$
    – Glen_b
    Commented Jan 23 at 17:44
  • $\begingroup$ To not worry much about overly conservative p-values is to be a classical $\alpha$-driven statistician (overstating it a bit to make a point). I want p-values that have minimum absolute errors. $\endgroup$
    – Frank Harrell
    Commented Jan 24 at 15:36
  • $\begingroup$ I would take issue with "This exposes the meaninglessness of significance testing in this context." It certainly might expose a lack of utility of dichotomising hypothesis tests, but an ordered listing of p-values from the significance tests will give a ranking of strength of evidence against the null hypotheses. It is the all-or-none aspect of significant/not significant that you should rail against, not the p-values. $\endgroup$
    – Michael Lew
    Commented Feb 14 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.