0
$\begingroup$

I have three datasets A, B and C. A and C can construct a contingency table, and I can get a p value (pValue1) with Fisher's exact test. B and C can construct another contingency table, and I can get another p value (pValue2) with Fisher's exact test.

I want to know how to combine these two p values and get a single one. One possible method is using Fisher's method, but it tends to generate a new value close to the minimum one. I want to use some other method to tend to generate one close to the maximum one, namely both reject the null hypothesis.

Can anyone tell me which method I can use?

$\endgroup$
  • $\begingroup$ p values are probabilities that something is equal to something. What are you wanting to know the probability of. Your question doesn't seem well-posed to me. $\endgroup$ – generic_user Mar 15 '13 at 20:48
  • $\begingroup$ I think you're asking about multiple hypothesis testing. In particular, you're probably most interested in the Holm-Bonferroni correction method wiki $\endgroup$ – user1448319 Mar 15 '13 at 23:02
3
$\begingroup$

Since it is evidently meaningful to form a contingency table from A and C, and from B and C, I suspect what you call "datasets" might be more commonly referred to as variables within a single dataset. Moreover, these seem to be categorical variables. I gather you want to test if there is an association between C and either A or B. You can determine that within a single model—that is, Fisher's exact test is inappropriate here and you don't need to combine p values. Instead, the appropriate analysis is a log-linear model.

Here is a quick R demonstration:

d = data.frame(A=    rep(c("yes","no"), each=40),
               B=rep(rep(c("yes","no"), each=20), 2),
               C=  c(rep(c("yes","no"), times=c( 5,15)),
                     rep(c("yes","no"), times=c(10,10)),
                     rep(c("yes","no"), times=c(10,10)),
                     rep(c("yes","no"), times=c(15, 5)) ) )
tab = table(d)
margin.table(tab, margin=c(1,3))
#      C
# A     no yes
#   no  15  25
#   yes 25  15
margin.table(tab, margin=c(2,3))
#      C
# B     no yes
#   no  15  25
#   yes 25  15
margin.table(tab, margin=c(1,2))
#      B
# A     no yes
#   no  20  20
#   yes 20  20

library(MASS)  # we use this package to get the loglm() function
loglm(~C + A*B, tab)
# Call:
# loglm(formula = ~C + A * B, data = tab)
# 
# Statistics:
#                       X^2 df   P(> X^2)
# Likelihood Ratio 10.46496  3 0.01500047
# Pearson          10.00000  3 0.01856614

I have a more extensive discussion here: $\chi^2$ of multidimensional data.

$\endgroup$
  • $\begingroup$ Just as @gung I would advise you to try log-linear models, especially if you have three way contingency tables. A good introduction, that captures $\chi^2$ , logistic regression and log-linear models is this pdf: mathdept.iut.ac.ir/sites/mathdept.iut.ac.ir/files/AGRESTI.PDF $\endgroup$ – user83346 Aug 24 '15 at 6:52
1
$\begingroup$

It seems you want to simultaneously test AvBvC in a 2x3 contingency table? It's not entirely clear what you are asking for.

You can calculate $\chi^2$ tests or Fisher Exact Tests (troublesome by hand) on tables larger than 2x2. For $\chi^2$ , the procedure is the same as in a 2x2 table - calculate the expected frequency of each cell as $$ E_{ij} = (RowTotal_i*ColumnTotal_j)/Total $$ Your $\chi^2$ statistic is then $$ \sum_{ij} {(O_{ij}-E_{ij})^2 \over E_{ij} }$$ with degrees of freedom $ (i-1)(j-1) $

Caveat here - your question makes it sound like you are fishing for a null rejection by throwing a battery of tests at your dataset. Recall that we reject the null if the p-value we obtain from a test is below an arbitrary point. By throwing multiple tests at a dataset, you increase by pure chance that you one of these tests will pass that arbitrary point. Thus this is generally frowned upon without correction for those multiple tests.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.