Comparison of statistical tests exploring co-dependence of two binary variables

Question

Suppose we observe data $(X_i,Y_i)_{i=1,...,n}$ on two binary variables: $X\in\{0,1\}$ and $Y\in\{0,1\}$. We would like to test if $X$ and $Y$ are co-dependent (related). Standard suggestions in mainstream textbooks are the following:

• chi-square test for independence of $X$ and $Y$,
• Z-test for comparing proportions of $[Y = 1]$ between two groups: $[X = 0]$ and $[X = 1]$,
• Z-test for comparing proportions of $[X = 1]$ between two groups: $[Y = 0]$ and $[Y = 1]$.

In addition to that, we can run logistic regressions of $Y$ on $X$ and $X$ on $Y$. We can check statistical significance of the slope coefficients. There are at least $3$ standard tests for that: likelihood ratio, Wald and deviance. Since we consider two regressions, there are $3 * 2 = 6$ tests added, making the total number $9$. But wait, we can run probit models too. Et cetera, et cetera, ...

Is there one or more references which systematically and rigorously answer(s) the following questions:

• Which tests are algebraically equivalent and when?
• Which tests are most powerful, why and when?
• What is the power function for each test and each sample size?
• In practical terms, which tests deliver the same verdict almost always?

Answer 1

Tan et al, in Information Systems 29 (2004) 293-313, consider 21 different measures for association patterns between 2 binary variables. Each of these measures has its strengths and weaknesses. As the authors state in the Abstract:

Objective measures such as support, confidence, interest factor, correlation, and entropy are often used to evaluate the interestingness of association patterns. However, in many situations, these measures may provide conflicting information about the interestingness of a pattern. ... In this paper, we describe several key properties one should examine in order to select the right measure for a given application. ... We show that depending on its properties, each measure is useful for some application, but not for others.

The major issues are in terms of the type of association in which one is interested and the properties of the measure that one wishes to maintain. For example, if P(X) is the probability of X = 1, P(Y) is the probability of Y = 1, and P(X,Y) is the probability that both are 1 in a 2 X 2 contingency table, which of the following matter to you in a measure of association:

• Is the measure 0 when X and Y are statistically independent?
• Does the measure increase with P(X,Y) as P(X) and P(Y) are constant?
• Does the measure decrease monotonically in either P(X) or P(Y) as the other probabilities remain constant?
• Is the measure symmetric under permutation of X and Y?
• Is it invariant to row and column scaling?
• Is it antisymmetric under row or column permutation?
• Is it invariant when both rows and columns are swapped?
• Is it invariant when extra cases in which both X and Y are 0 are added?

No measure has all of these properties. The issue of which type of measure makes the most sense in a particular application thus would seem to be more crucial than generic considerations of power; the "verdict" might well depend on the type of association of interest.

If your measure of codependence between X and Y is the type examined by a $\chi^2$ test of a 2 x 2 contingency table, then this answer also answers your question with respect to logistic regression and chi-square tests. Briefly:

• Asymptotically, all the logistic regression tests are equivalent. The likelihood-ratio test is based on deviance so they're not 2 separate tests. The score test for a logistic regression (not mentioned in your question) is exactly equivalent to a $\chi^2$ test without continuity correction. With large numbers of cases, when the underlying assumptions of normality hold, they should all provide the same results.

• For a logistic regression model, the likelihood-ratio test is generally preferred. The Wald test assumes a symmetric, normal distribution of the log-likelihood profile around the maximum-likelihood estimate that might not be found in a small sample. The score test in general is less powerful in practice. (I haven't worked through the details specific to a 2-way contingency table, however.) Power functions would typically be calculated based on the assumptions underlying the tests (effectively normality assumptions, as also underly the separate F-tests you note), which suggests to me that they would be the same under those assumptions. In practical applications involving small numbers of cases, such theoretical power functions might be misleading.

In answering this question, I assumed that the observations on variables X and Y are not paired. If they are paired then see the answer from @Alexis.

Answer 2

The z test of proportions assumes your samples are independent (your single index $i$ for both $X$ and $Y$ and your indication of a regression context implies that these are paired, not independent observations).

Therefore, McNemar's test (1947) is an appropriate test for association in paired binary data:

The positivist null hypothesis is that $X$ and $Y$ are not associated, with the positivist alternative being that $X$ and $Y$ are associated. One way of expressing this is:

$$H_{0}{+}: P(Y=1|X=0) = P(Y=1|X=1)\text{, and}$$ $$H_{\text{A}}^{+}: P(Y=1|X=0) \ne P(Y=1|X=1)$$

Another way of expressing it is:

$$H_{0}{+}: P(Y=1|X=0) = P(X=1|Y=0)\text{, and}$$ $$H_{\text{A}}^{+}: P(Y=1|X=0) \ne P(X=1|Y=0)$$

There are still other ways (e.g., using odds ratios, etc.)

McNemar's test uses counts of pairs:

• $X=0, Y=0$ (concordant pair)
• $X=1, Y=0$ (discordant pair, call this count $r$)
• $X=0, Y=1$ (discordant pair, call this count $s$)
• $X=1, Y=1$ (concordant pair)

And specifically, McNemar's test only uses counts of discordant pairs (i.e. $r$ and $s$) to construct it's test statistic (the below formulation includes a continuity correction (i.e. counts are discrete by definition, but $\chi^{2}$ distribution is actually continuous):

$$\chi^{2} = \frac{|(r-s)|-1^{2}}{r+s}$$

This test statistic has a single degree of freedom, and $p = P(X^{2}_{\nu=1} > \chi^{2})$.

Per Bennett and Underwood (1970) "The McNemar test is in fact UMP for $p'=\frac{1}{2}$ against alternatives $p'\ne\frac{1}{2}$ by an argument similar to that of Lehmann ([1959], section 4.7)."

McNemar's test expresses the degree of association using an odds ratio $= \frac{r}{s}$, where $OR=1$ is consistent with no association, $OR>1$ are consistent with the odds of $Y=1$ being greater for $X=1$ relative to $X=0$, and vice versa. The confidence interval for this OR is $e^{\ln (\frac{r}{s})-z_{\alpha/2}\sqrt{(r+s)/rs}}$, $e^{\ln (\frac{r}{s})+z_{\alpha/2}\sqrt{(r+s)/rs}}$.

McNemar's test can also be framed as a test for equivalence with the negativist null hypothesis that $X$ and $Y$ are associated by at least $\Delta$ (your equivalence threshold, aka the relevant association you care about). $\Delta$ in this application takes values between 0 (zero association) and 1 (perfect association). The alternative hypothesis is that $X$ and $Y$ do not have an association as strong as $\Delta$ or stronger.

Interestingly, the test statistics for McNemar's equivalence test (in the TOST framework) are z distributed, not $\chi^{2}$ distributed (Liu, et all, 2002):

$z_{1} = \frac{n\Delta - [(r-s) - 1]}{\sqrt{(r+s)-n(\frac{r}{n}-\frac{s}{n})^{2}}}$, and
$z_{2} = \frac{[(r+s)+1]+n\Delta}{\sqrt{(r+s)-n(\frac{r}{n}-\frac{s}{n})^{2}}}$

These statistics have been constructed for upper tail rejection regions:

$p_{1} = P(Z > z_{1})$, and
$p_{2} = P(Z > z_{2})$

Only if both $p_{1} \le \alpha$ and $p_{2} \le \alpha$ can you reject the negativist null hypothesis, and conclude equivalence.

References
Bennett, B. M., & Underwood, R. E. (1970). 283. Note: On McNemar’s Test for the 2 $\times$ 2 Table and Its Power Function. Biometrics, 26(2), 339–343.

Liu, J., Hsueh, H., Hsieh, E., & Chen, J. J. (2002). Tests for equivalence or non-inferiority for paired binary data. Statistics In Medicine, 21, 231–245.

McNemar, Q. (1947). Note on the Sampling Error of the Difference Between Two Correlated Proportions or Percentages. Psychometrika, 12(2), 153–157.

Answer 3

To me, this seems like a case where you should use Fisher's exact test.

I.e. you would make a table of your outcomes,

╔═════╦═════════════╦═════════════╗
║     ║     X=0     ║     X=1     ║
╠═════╬═════════════╬═════════════╣
║ Y=0 ║ |[X=0^Y=0]| ║ |[X=1^Y=0]| ║
║ Y=1 ║ |[X=0^Y=1]| ║ |[X=1^Y=1]| ║
╚═════╩═════════════╩═════════════╝

where e.g. |[X=0^Y=0]| are the number of data points with $X_i$=0 and $Y_i$=0.

As explained in the wikipedia entry, the values in the table will follow a hypergeometric distribution.