7
$\begingroup$

In a 2x2 analysis of categorical data, what causes the p value for the test of the likelihood ratio to differ from both the Fisher's Exact Test and the test of linear-by-linear Association? Alternatively, where can I go to learn this? (I am using SPSS.) For example, in one analysis, the Fisher's Exact test 2 sided is .051 and 1 sided is .027, the Likelihood ratio test p is .031 and the linear-by-linear asssocaition p is .039. Some of these p values are less than .05, some more. Which do I trust, or how do I choose from among these analyses?

$\endgroup$
  • 4
    $\begingroup$ Flip the question around -- why should the p-values be the same? They're not equivalent tests. $\endgroup$ – Glen_b -Reinstate Monica Nov 30 '14 at 5:59
  • $\begingroup$ How do the tests differ? Is the null hypothesis for each the same? Are the assumptions the same? Alternatively, where can I go to learn about this? $\endgroup$ – Joel W. Nov 30 '14 at 13:47
  • 2
    $\begingroup$ The test statistics differ and are not equally sensitive to exactly the same sample arrangements. If they were the same, they'd be utterly redundant - effectively the same test, and would perhaps appear together in a hyphenated name (as with Mann-Whitney-Wilcoxon, because the Wilcoxon rank sum test statistics is effectively the Mann-Whitney U test statistic plus a constant, so they are equivalent tests). So again, why should the p-values be the same? $\endgroup$ – Glen_b -Reinstate Monica Dec 1 '14 at 2:35
10
+50
$\begingroup$

You have a few issues here. First, understanding what each test is doing, and second interpreting the p-values.

First, each test has different underlying assumptions. The likelihood ratio test statistic is formed by taking the log of the ratio of the likelihood under the null model, divided by the alternative model. The test statistic is approximately chi-squared distributed, and is asymptotically equivalent to the Pearson Chi-squared test. Because you are calculating p-values using an asymptotic approximation, you want to make sure you have enough data to justify doing this. Further, one of the key assumptions of this test is that the observations are independent.

Fisher's Exact test, as its name implies, calculates an exact p-value based on the underlying assumptions. Instead of using a continuous distribution as an approximation as the sample size grows, it is based on a discrete distribution. Specifically, the probability of observing any 2x2 table follows the hypergeometric distribution. This test is typically used when there are not enough observations to justify the assumptions of the asymptotic tests, although you could use it for any given number of observations (see edit below, I no longer believe this to be correct). Importantly however, this test still assumes independence. For both this test and the likelihood ratio test, your null hypothesis is that the probability of each outcome is equal.

The linear-by-linear association model is testing something different. This models the log odds ratio as a function of each variable, as well as a term accounting for the relationship between between the variables. You are estimating a general linear model, assuming a Poisson distribution of counts with the log link function. I am not familiar with SPSS, but the significance of that test may indicate that your observations are not independent (and thus violate the assumptions of the other tests). (edit: I think you can ignore this issue based on what SPSS output).

Finally, it seems like you have some issues interpreting your p-values. It might be worth considering what the difference between a p-value of 0.051 vs. 0.049 means practically. Do you consider the first to be significantly different than the second in terms of evidence it provides? Another issue you may want to investigate is calculating mid p-values. This can help account for some conservatism of tests based on discrete distributions. For example, say one more observation would change your Fisher exact p-value from 0.051 to 0.025. This discontinuity in p-values can effectively make tests like the Fisher exact test more conservative. For a reference on all of these topics, I would recommend Categorical Data Analysis by Agresti.

Edit: I'll address a few additional points more in depth. 1) Why are the p-values different and 2) Which test to use

To start at the top, the p-values in general for these tests can be different because they are using different assumptions. To illustrate, I generated some random sample data for 2x2 tables. To do this I started with n = 10 (5 data points in each row), and went to n = 1000. Row 1 had a true probability of 30%, and row 2 had a true probability of 70%. Because we know that the odds ratios are truly different, we should ideally see a low p-value. The chart below shows the difference between the p-value estimated by fisher's test, vs. the Pearson chi-square test (this was easier for me to run quickly to illustrate the point than the LR-test).

p value illustration

Note that, especially for small n, these values can be very different although they differ less when a continuity correction is applied.

Second, which test should you use? Now that you have posted your actual data in the comments, I believe you should use Fisher's test. This is because you have a zero cell. However, because this test is conservative, you should probably apply a correction (mid-P corrections are what I am familiar with, not sure if there are other superior options). See the thread below for more discussion and references. That thread also caused me to reconsider suggesting that Fisher's test could be used in any situation, given the evidence the author provides:

Given the power of computers these days, is there ever a reason to do a chi-squared test rather than Fisher's exact test?

Finally, this site suggests you can ignore the issue of the linear-by-linear test. I didn't go into too much detail, but it seems that it may be equivalent in SPSS to the Chi-Square test: https://sites.google.com/a/lakeheadu.ca/bweaver/Home/statistics/notes/chisqr_assumptions

$\endgroup$
  • $\begingroup$ I just ran a 2x2 test and got p=.026 for the likelihood ratio and p=.111 for the Fisher Exact Test (2 tailed). These ps are very different. The cells are: row one 8, 0, and row two 79, 30. What about the assumptions or the null hypotheses or the approaches taken explains this large difference. Which test do I trust? How do the null hypotheses differ? (Fisher 1 tailed p=.086.) $\endgroup$ – Joel W. Dec 3 '14 at 21:18
  • $\begingroup$ Please you explain your comment: "observations are not independent (and thus violate the assumptions of the other tests)." Independence of observations is something I evaluate based on the experimental design. You must be referring to something else. Please clarify. $\endgroup$ – Joel W. Dec 3 '14 at 21:22
  • $\begingroup$ The continuity corrected chi sq = 1.69, p=.193 (for the 2x2 example in the comment above). Does this pattern of p values make sense. Should I expect this pattern with other 2x2 analyses or is there something unusual about the data sets I have tried? $\endgroup$ – Joel W. Dec 3 '14 at 22:34
  • $\begingroup$ For your first comment: a general rule of thumb is that you should have no zero cell counts to use the asymptotic tests. In this case you should use the Fisher test. In fact, as stated above, assuming assumptions are correct you are not wrong using the Fisher test over LR for any 2x2. For your second comment, I think you are misinterpreting what I am saying. Since I am not familiar with SPSS, I was not certain what that output signified. I am merely describing what a linear-by-linear model is, and suggesting you may want to investigate further. $\endgroup$ – Eric Brady Dec 3 '14 at 23:19
  • $\begingroup$ The LR test yields a p.026 and the Fisher Exact Test yields a p=.111. You say I am not wrong using the Fisher. Am I wrong using the L.R? Why are they yielding such different p's? Does one test have more power than the other? Are they testing different null hypotheses? If they are testing the same null hypothesis, is one approach uniformly the most powerful? $\endgroup$ – Joel W. Dec 4 '14 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.