Suppose we observe data $(X_i,Y_i)_{i=1,...,n}$ on two binary variables: $X\in\{0,1\}$ and $Y\in\{0,1\}$. We would like to test if $X$ and $Y$ are co-dependent (related). Standard suggestions in mainstream textbooks are the following:

  • chi-square test for independence of $X$ and $Y$,
  • Z-test for comparing proportions of $[Y = 1]$ between two groups: $[X = 0]$ and $[X = 1]$,
  • Z-test for comparing proportions of $[X = 1]$ between two groups: $[Y = 0]$ and $[Y = 1]$.

In addition to that, we can run logistic regressions of $Y$ on $X$ and $X$ on $Y$. We can check statistical significance of the slope coefficients. There are at least $3$ standard tests for that: likelihood ratio, Wald and deviance. Since we consider two regressions, there are $3 * 2 = 6$ tests added, making the total number $9$. But wait, we can run probit models too. Et cetera, et cetera, ...

Is there one or more references which systematically and rigorously answer(s) the following questions:

  • Which tests are algebraically equivalent and when?
  • Which tests are most powerful, why and when?
  • What is the power function for each test and each sample size?
  • In practical terms, which tests deliver the same verdict almost always?
  • 1
    $\begingroup$ also I remember logistic regression and test being equivalent to z test on log-odds, which should be the same as chi square test. but I'm not sure of what kind on test on the logistic model, most probably wald one $\endgroup$ – carlo Nov 20 '19 at 13:56
  • $\begingroup$ I don't know such literature, but I could offer to do my own analysis with a simulation: It will reveal if some of these methods (always) give identical outputs. Afterwards, I should have the mathematical intuition to argue why the ones, which are different, behave like they do. $\endgroup$ – KaPy3141 Nov 26 '19 at 11:32
  • 1
    $\begingroup$ Thank you for the offer. Simulation is always a possibility. However, accurate simulation on a big grid of parameter values and sample sizes would be a sizable task. Even then it would not necessarily provide intuition comparable to formulas... A good part of the problem must have been solved analytically somewhere. For the rest, there must have been extensive simulation studies over the previous decades. $\endgroup$ – stans - Reinstate Monica Nov 26 '19 at 16:11

To me, this seems like a case where you should use Fisher's exact test.

I.e. you would make a table of your outcomes,

║     ║     X=0     ║     X=1     ║
║ Y=0 ║ |[X=0^Y=0]| ║ |[X=1^Y=0]| ║
║ Y=1 ║ |[X=0^Y=1]| ║ |[X=1^Y=1]| ║

where e.g. |[X=0^Y=0]| are the number of data points with $X_i$=0 and $Y_i$=0.

As explained in the wikipedia entry, the values in the table will follow a hypergeometric distribution.

  • $\begingroup$ Could you elaborate please? $\endgroup$ – Michael R. Chernick Dec 18 '19 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.