1
$\begingroup$

I asked this question on Stack Overflow too but I'm not sure that's the most appropriate place since it might actually be a statistical issue and not an R issue. Sorry if cross posting is not allowed.

I have contingency tables where I'm using a response variable with 15 categories combined with predictor variables with similar numbers of categories (e.g. predictor1 x response, predictor2 x response, etc.). I'm using R and running fisher.test to test for independence and, regardless of which predictor I use, I get the same exact p-value. I'm using Fisher's exact test because I have many expected values less than 1 and I'm using the simulate.p.value=TRUE parameter in R because I get a workspace size error otherwise. I've tried increasing the B parameter, which is the number of simulations that are run, to various values, and that changes the p-values, but they're all changed exactly the same amount and so remain identical. I'm even getting this p-value indicating statistical significance for predictor variables which are absolutely not statistically significant, such as when I use the person who coded the sample as the predictor variable.

This problem doesn't occur if I draw up a small contingency table in R. The following two tests produce different p-values:

fisher.test(table(c("joe", "joe", "lou", "lou", "lou", "jill"), c("red", "green", "green", "blue", "blue", "blue")))
fisher.test(table(c("joe", "joe", "lou", "lou", "lou", "jill", "jill"), c("red", "green", "green", "blue", "blue", "blue", "red")))

Then here's an actual table from my data:

         kek lawl  lel lmao lmfao  lol  lul  mdr ptdr rjlol rofl roflmao rotflmao topkek trolol
    173   7    0    5    18    13  205   0    1    1     0    1       0        0      0      0
    302   3    0    0   190    15 1112   2    0    0     1    4       1        0      1      0
    322   0    0    0     0     0    0   0    1    0     0    0       0        0      0      0
    572   0    3    0    30    14  433   2    0    0     0    1       0        0      0      1
    756   0    0    0     4     3   33   0   19   16     0    0       0        0      0      0
    799   0    0    0     0     0    4   0   36    0     0    0       0        0      0      0
   1032   4    1    0   115    18 1251   3    0    0     0    1       1        2      0      0
   1097   0    0    0    16     0  175   0    0    0     0    0       0        0      0      0
   1227   0    0    0    15     5  233   0    0    0     0    0       0        0      0      0
   1291   0    0    0    11     0   96   0   11    0     0    0       0        0      0      0
   1340   0    0    0     0     0    0   0   23    0     0    0       0        0      0      0
   1782   0    0    0     0     0    0   0    0    1     0    0       0        0      0      0
   1917   0    0    1    88    31  261   0    5    0     0    0       0        0      0       
   2067   0    0    0     0     0    7   0   16    8     0    0       0        0      0      0
   2265   0    0    0     7     1   28   2    3    1     0    0       0        0      0      1
   2305   0    0    0     0     0    0   0    1    0     0    0       0        0      0      0
   6445   0    0    0     0     0    0   0    1    0     0    0       0        0      0      0
   6744   0    0    0     0     0    0   0    3    0     0    0       0        0      0      0
   6817   0    0    0     0     0   18   0    3    0     0    0       0        0      0      0

When I run the following code on it:

fisher.test(table(adjoint$communaute, adjoint$lol), simulate.p.value=TRUE)

I get:

Fisher's Exact Test for Count Data with simulated p-value (based on 2000 replicates)

data:  table(adjoint$communaute, adjoint$lol)
p-value = 0.0004998
alternative hypothesis: two.sided

If I do the test for the table below, I get the same exact result:

         kek lawl  lel lmao lmfao  lol  lul  mdr ptdr rjlol rofl roflmao rotflmao topkek trolol
  Josh     3    3    0   22     1  227    0   25    0     0    0       0        0      0      0
  Shari   11    1    6  472    99 3629    9   98   27     1    7       2        2      1      2

There's no way that an association exists between the two variables in the second table because the rows indicate the person who coded the sample being used and the columns indicate what someone wrote on Twitter to mean "lol".

Why would I get the same p-value with this test no matter which variable I use for the rows in the contingency table?

$\endgroup$
  • 1
    $\begingroup$ Just to clarify -- the Fisher test doesn't use a predictor variable, it tests independence in a contingency table. There's no predictor and response, it's just two factors. In R's implementation in stats::fisher.test, there are two ways of supplying the information corresponding to the contingency table. The first is to supply the contingency table itself (an $r\times c$ table of counts). The second is to supply two factor variables (corresponding to the particular row and column in an $r\times c$ table); this is then converted to an $r\times c$ table in the function fisher.test by calling $\endgroup$ – Glen_b Sep 9 '17 at 0:58
2
$\begingroup$
  1. The sameness of the p-values may be an artifact of the number of replicates used in the simulation. You might add a B=5000 or B=10000 option to fisher.test.

  2. I'm not seeing what you're seeing. For example if I compare Josh to Shari, I don't get the same p-value as if I compare 1032 to 1093 or 6744 to 6817 (p < 0.0001, p = 0.8, p = 0.01, respectively).

  3. "There's no way that an association exists." I wonder if you are thinking about the hypothesis being tested correctly. For each word, the proportions seen in Josh or Shari vary some. I also wonder if you're relying too heavily on the p-value, and ignoring an effect size or looking at the odds ratios.

`

 Input =("
 Name    kek lawl  lel lmao lmfao  lol  lul  mdr ptdr rjlol rofl roflmao rotflmao topkek trolol
 Josh     3    3    0   22     1  227    0   25    0     0    0       0        0      0      0
 Shari   11    1    6  472    99 3629    9   98   27     1    7       2        2      1      2
 ")
 Matrix = as.matrix(read.table(textConnection(Input),
                   header=TRUE,
                   row.names=1))

Matrix

fisher.test(Matrix, simulate.p.value=TRUE, B=10000)

prop.table(Matrix, margin=1)

library(vcd)

assocstats(Matrix)

loddsratio(Matrix)
$\endgroup$
  • $\begingroup$ Thanks for the reply. Unfortunately, I've tried all the way up to B=2000000 and the p-value is always identical from one predictor variable to the other.In the first table, the numbers are ID numbers for groups of people who have some common $\endgroup$ – joshisanonymous Sep 8 '17 at 22:04
  • $\begingroup$ ... something in common, so I'm not quite sure what you mean by comparing 1032 to 1093 or 6744 to 6817 because the variable is "group ID". I was trying to test if there's an association between group and whether one uses lol, lmao, etc. $\endgroup$ – joshisanonymous Sep 8 '17 at 22:11
  • 1
    $\begingroup$ ... I was going off of your example of Josh and Shari... When I make a 2 x c table, like with Josh and Shari or with 1032 and 1097, I get different p-values. Are you not seeing the same? $\endgroup$ – Sal Mangiafico Sep 9 '17 at 0:03
  • $\begingroup$ If I compare 1032 to 1097 only, and use B=10000, I get a p = 0.7946, but if I use the whole table and use your exact code just changing the Input, I get p = 9.999e-05, which is the same p-value that I get when I use your code to with Josh/Shari. Is this just an issue of having too many samples, perhaps? $\endgroup$ – joshisanonymous Sep 9 '17 at 15:38
  • $\begingroup$ I'm not sure. I suspect it has something to do with the procedure mathematically, and maybe the fact that your matrix is so sparse. $\endgroup$ – Sal Mangiafico Sep 9 '17 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.