4
$\begingroup$

When does/can one use the likelihood ratio significance test instead of Fisher's exact test or its Pearson $\chi^2$ approximation for comparing two binomial datasets?

Given two binomial datasets (distributions), I'm seeing the LR test being used to compare one distribution against the global (combined) distribution. Usually I apply Fisher's test for comparing one dataset against the other. I realize that LR testing is of the Neyman-Pearson school, which assumes a fully specified alternative model as well as null model. E.g., in the LR test Wikipedia page example, it's being used to compare two binomial datasets (# heads/tails for two coins).

Why not use the $\chi^2$ test to compare the two samples against each other? What are the conceptual differences in these two approaches? When do I use which? And when is it appropriate to compare one sample against not its complement but the global dataset?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Generally speaking, the likelihood ratio and the ordinary Pearson $\chi^2$ tests are more accurate than Fisher's "exact" test. But for your situation you need an extremely heavy multiplicity adjustment thrown in, not matter which statistical test is used. Decision trees such as the one you are building require amazingly large datasets for their structure to validate. In a quick look at the CN2 link you provided I could not tell if the algorithms incorporated shrinkage (panelization; regularization). If not, watch for over-interpretation.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Could you elaborate on the relative merits/appropriateness of LR vs. Pearson $\chi^2$ (which is just an approximation of Fisher's exact test)? And why are these more accurate than Fisher's exact test? $\endgroup$
    – Yang
    May 4, 2011 at 19:39
  • $\begingroup$ Edited my question to remove the rule mining context - I wasn't sure if that would be helpful, but I think it's distracting from my main question. BTW, it's not a decision tree algorithm - rule mining is a different problem, and it tries to output only the most significant rules, so over-interpretation is less of an issue. (I believe.) $\endgroup$
    – Yang
    May 4, 2011 at 19:48
  • 2
    $\begingroup$ Nothing in what you wrote provides any indication that the method is safe from overinterpretation (a form of overfitting). One reason I worry about the method is that it doesn't treat continuous variables continuously. The problem with Fisher's test is that the P-values are too large. See for example Crans GG Stat in Med 27:3598; 2008. The word "exact" has misled practitioners for decades. The ordinary Pearson $\chi^2$ is usually more accurate. $\endgroup$
    – Frank Harrell
    May 4, 2011 at 23:57
  • $\begingroup$ Interesting - sounds like a relatively recent result that directly contradicts I think several sources I've come across over time that recommend using Fisher over $\chi^2$. Unfortunately I don't have access to that paper. What about the likelihood ratio test, and what about comparing one dataset against the combined dataset - when should I use these? $\endgroup$
    – Yang
    May 5, 2011 at 1:01
  • $\begingroup$ The main result of the paper:`the test size of FET [when testing at the 0.05 level] was less than 0.035 for nearly all sample sizes before 50 and did not approach 0.05 even for sample sizes over 100' I don't have a ready reference on the LR test but I expect it would perform similar to Pearson's test, and be better than FET. Please elaborate on the original global purpose of the analysis. $\endgroup$
    – Frank Harrell
    May 5, 2011 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.