# Comparison of statistical tests exploring co-dependence of two binary variables

Suppose we observe data $$(X_i,Y_i)_{i=1,...,n}$$ on two binary variables: $$X\in\{0,1\}$$ and $$Y\in\{0,1\}$$. We would like to test if $$X$$ and $$Y$$ are co-dependent (related). Standard suggestions in mainstream textbooks are the following:

• chi-square test for independence of $$X$$ and $$Y$$,
• Z-test for comparing proportions of $$[Y = 1]$$ between two groups: $$[X = 0]$$ and $$[X = 1]$$,
• Z-test for comparing proportions of $$[X = 1]$$ between two groups: $$[Y = 0]$$ and $$[Y = 1]$$.

In addition to that, we can run logistic regressions of $$Y$$ on $$X$$ and $$X$$ on $$Y$$. We can check statistical significance of the slope coefficients. There are at least $$3$$ standard tests for that: likelihood ratio, Wald and deviance. Since we consider two regressions, there are $$3 * 2 = 6$$ tests added, making the total number $$9$$. But wait, we can run probit models too. Et cetera, et cetera, ...

Is there one or more references which systematically and rigorously answer(s) the following questions:

• Which tests are algebraically equivalent and when?
• Which tests are most powerful, why and when?
• What is the power function for each test and each sample size?
• In practical terms, which tests deliver the same verdict almost always?
• also I remember logistic regression and test being equivalent to z test on log-odds, which should be the same as chi square test. but I'm not sure of what kind on test on the logistic model, most probably wald one Nov 20 '19 at 13:56
• I don't know such literature, but I could offer to do my own analysis with a simulation: It will reveal if some of these methods (always) give identical outputs. Afterwards, I should have the mathematical intuition to argue why the ones, which are different, behave like they do. Nov 26 '19 at 11:32
• Thank you for the offer. Simulation is always a possibility. However, accurate simulation on a big grid of parameter values and sample sizes would be a sizable task. Even then it would not necessarily provide intuition comparable to formulas... A good part of the problem must have been solved analytically somewhere. For the rest, there must have been extensive simulation studies over the previous decades. Nov 26 '19 at 16:11
• If you had continuous data (say for both $X$ and $Y$), would you just run a correlation test (or Hoeffding's test for more general dependence than linear Pearson or monotonic Spearman correlation)? (I think the answer is that you would.) If so, then I think this comes down to examining the copula between the two discrete variables and checking if it is significantly different from $U((0,1)\times(0,1))$.
– Dave
Jan 28 '20 at 17:12

Tan et al, in Information Systems 29 (2004) 293-313, consider 21 different measures for association patterns between 2 binary variables. Each of these measures has its strengths and weaknesses. As the authors state in the Abstract:

Objective measures such as support, confidence, interest factor, correlation, and entropy are often used to evaluate the interestingness of association patterns. However, in many situations, these measures may provide conflicting information about the interestingness of a pattern. ... In this paper, we describe several key properties one should examine in order to select the right measure for a given application. ... We show that depending on its properties, each measure is useful for some application, but not for others.

The major issues are in terms of the type of association in which one is interested and the properties of the measure that one wishes to maintain. For example, if P(X) is the probability of X = 1, P(Y) is the probability of Y = 1, and P(X,Y) is the probability that both are 1 in a 2 X 2 contingency table, which of the following matter to you in a measure of association:

• Is the measure 0 when X and Y are statistically independent?
• Does the measure increase with P(X,Y) as P(X) and P(Y) are constant?
• Does the measure decrease monotonically in either P(X) or P(Y) as the other probabilities remain constant?
• Is the measure symmetric under permutation of X and Y?
• Is it invariant to row and column scaling?
• Is it antisymmetric under row or column permutation?
• Is it invariant when both rows and columns are swapped?
• Is it invariant when extra cases in which both X and Y are 0 are added?

No measure has all of these properties. The issue of which type of measure makes the most sense in a particular application thus would seem to be more crucial than generic considerations of power; the "verdict" might well depend on the type of association of interest.

If your measure of codependence between X and Y is the type examined by a $$\chi^2$$ test of a 2 x 2 contingency table, then this answer also answers your question with respect to logistic regression and chi-square tests. Briefly:

• Asymptotically, all the logistic regression tests are equivalent. The likelihood-ratio test is based on deviance so they're not 2 separate tests. The score test for a logistic regression (not mentioned in your question) is exactly equivalent to a $$\chi^2$$ test without continuity correction. With large numbers of cases, when the underlying assumptions of normality hold, they should all provide the same results.

• For a logistic regression model, the likelihood-ratio test is generally preferred. The Wald test assumes a symmetric, normal distribution of the log-likelihood profile around the maximum-likelihood estimate that might not be found in a small sample. The score test in general is less powerful in practice. (I haven't worked through the details specific to a 2-way contingency table, however.) Power functions would typically be calculated based on the assumptions underlying the tests (effectively normality assumptions, as also underly the separate F-tests you note), which suggests to me that they would be the same under those assumptions. In practical applications involving small numbers of cases, such theoretical power functions might be misleading.

In answering this question, I assumed that the observations on variables X and Y are not paired. If they are paired then see the answer from @Alexis.

• Thank you for the reference. The paper is interesting in itself. However, it does not answer my exact questions. Instead of explaining how to detect codependent relationships $(X,Y)$ with highest probability, the paper is essentially saying: "Suppose we restrict ourselves only to relationships which are codependent, at least mildly. How do we rank them? How do we systematize numerous metrics for quantifying the strength of the relationship?"... In your post you are stating: ... Feb 2 '20 at 15:01
• ... "The issue of which type of measure makes the most sense in a particular application thus would seem to be more crucial than generic considerations of power..." I politely disagree. I need the power. When deciding between hypotheses [H0: $X$ and $Y$ are independent] and [H1: $X$ and $Y$ are codependent] I need to detect H1 with highest probability when H1 is true... In many parts of finance and other fields, one needs to prescreen the signals first. And then one enters relevant signals into various measures and models, potentially even more flexible than the measures reviewed in Tan et. al Feb 2 '20 at 15:01
• @stans-ReinstateMonica my point is that to decide whether X and Y are codependent, you need to start with your desired definition of codependence. I've added a couple of paragraphs with respect to contingency-table analysis as done with chi-square or logistic regression.
– EdM
Feb 3 '20 at 17:09
• Thank you for the elaboration. Sadly, asymptotically we all are dead. Yes, the likelihood ratio test is most statistically efficient asymptotically, under certain regularity assumptions, but how does it perform on samples of 30, 100, 200? Kind of samples I work with every day as a statistician. You see, I have not noticed that "the likelihood-ratio test is generally preferred" to Wald test for logistic regression. SPSS and Stata display Wald test for each coefficient as the default. They default to using LR only as the overall, omnibus test for the model... Feb 5 '20 at 11:37
• ... Both Wald test and LR test make distributional assumptions. Wald test says that the relevant statistic is Gaussian while LR test says that the LR statistic has chi-square distribution. The latter statement is not necessarily true in small and medium-sized samples. Here I am talking about the popular version of the LR test, based on the chi-square approximation and implemented in all statistical packages. I am not referring to the most general set-up by Neyman.... My main question: which test is most powerful in real life? I define codependence as $P(X = 1, Y =1) \neq P(X = 1)P(Y =1)$. Feb 5 '20 at 11:44

The z test of proportions assumes your samples are independent (your single index $$i$$ for both $$X$$ and $$Y$$ and your indication of a regression context implies that these are paired, not independent observations).

Therefore, McNemar's test (1947) is an appropriate test for association in paired binary data:

The positivist null hypothesis is that $$X$$ and $$Y$$ are not associated, with the positivist alternative being that $$X$$ and $$Y$$ are associated. One way of expressing this is:

$$H_{0}{+}: P(Y=1|X=0) = P(Y=1|X=1)\text{, and}$$ $$H_{\text{A}}^{+}: P(Y=1|X=0) \ne P(Y=1|X=1)$$

Another way of expressing it is:

$$H_{0}{+}: P(Y=1|X=0) = P(X=1|Y=0)\text{, and}$$ $$H_{\text{A}}^{+}: P(Y=1|X=0) \ne P(X=1|Y=0)$$

There are still other ways (e.g., using odds ratios, etc.)

McNemar's test uses counts of pairs:

• $$X=0, Y=0$$ (concordant pair)
• $$X=1, Y=0$$ (discordant pair, call this count $$r$$)
• $$X=0, Y=1$$ (discordant pair, call this count $$s$$)
• $$X=1, Y=1$$ (concordant pair)

And specifically, McNemar's test only uses counts of discordant pairs (i.e. $$r$$ and $$s$$) to construct it's test statistic (the below formulation includes a continuity correction (i.e. counts are discrete by definition, but $$\chi^{2}$$ distribution is actually continuous):

$$\chi^{2} = \frac{|(r-s)|-1^{2}}{r+s}$$

This test statistic has a single degree of freedom, and $$p = P(X^{2}_{\nu=1} > \chi^{2})$$.

Per Bennett and Underwood (1970) "The McNemar test is in fact UMP for $$p'=\frac{1}{2}$$ against alternatives $$p'\ne\frac{1}{2}$$ by an argument similar to that of Lehmann ([1959], section 4.7)."

McNemar's test expresses the degree of association using an odds ratio $$= \frac{r}{s}$$, where $$OR=1$$ is consistent with no association, $$OR>1$$ are consistent with the odds of $$Y=1$$ being greater for $$X=1$$ relative to $$X=0$$, and vice versa. The confidence interval for this OR is $$e^{\ln (\frac{r}{s})-z_{\alpha/2}\sqrt{(r+s)/rs}}$$, $$e^{\ln (\frac{r}{s})+z_{\alpha/2}\sqrt{(r+s)/rs}}$$.

McNemar's test can also be framed as a test for equivalence with the negativist null hypothesis that $$X$$ and $$Y$$ are associated by at least $$\Delta$$ (your equivalence threshold, aka the relevant association you care about). $$\Delta$$ in this application takes values between 0 (zero association) and 1 (perfect association). The alternative hypothesis is that $$X$$ and $$Y$$ do not have an association as strong as $$\Delta$$ or stronger.

Interestingly, the test statistics for McNemar's equivalence test (in the TOST framework) are z distributed, not $$\chi^{2}$$ distributed (Liu, et all, 2002):

$$z_{1} = \frac{n\Delta - [(r-s) - 1]}{\sqrt{(r+s)-n(\frac{r}{n}-\frac{s}{n})^{2}}}$$, and
$$z_{2} = \frac{[(r+s)+1]+n\Delta}{\sqrt{(r+s)-n(\frac{r}{n}-\frac{s}{n})^{2}}}$$

These statistics have been constructed for upper tail rejection regions:

$$p_{1} = P(Z > z_{1})$$, and
$$p_{2} = P(Z > z_{2})$$

Only if both $$p_{1} \le \alpha$$ and $$p_{2} \le \alpha$$ can you reject the negativist null hypothesis, and conclude equivalence.

References
Bennett, B. M., & Underwood, R. E. (1970). 283. Note: On McNemar’s Test for the 2 $$\times$$ 2 Table and Its Power Function. Biometrics, 26(2), 339–343.

Liu, J., Hsueh, H., Hsieh, E., & Chen, J. J. (2002). Tests for equivalence or non-inferiority for paired binary data. Statistics In Medicine, 21, 231–245.

McNemar, Q. (1947). Note on the Sampling Error of the Difference Between Two Correlated Proportions or Percentages. Psychometrika, 12(2), 153–157.

• I didn't interpret the question as representing paired observations, but if your interpretation is correct so is your answer (+1). Is there a reference with respect to the power of McNemar's test, given that power seems to be the main interest here?
– EdM
Feb 3 '20 at 17:29
• @EdM Updated with a note on power. The OP should really clarify explicitly whether these are paired or independent observations. Feb 3 '20 at 18:19
• Either way, there seems to be an answer provided on this page now. Maybe more useful to future visitors to this site if the ambiguity isn't resolved? At the end of my answer I added its restriction to unpaired data.
– EdM
Feb 3 '20 at 18:37
• $X_i$ is paired with $Y_i$ in observation $i$. That is the whole reason why we are interested in whether $X$ and $Y$ are codependent. For example, $X_i$ can be the gender of subject $i$ and $Y_i$ can be the indicator of whether he/she has ever become a CEO/owner of something. However, Z-test for proportions does not assume that $X_i$ and $Y_i$ are independent. It assumes that $(X_i, Y_i)$ is independent of $(X_j, Y_j)$. Z-test for proportions can be used to compare the proportion of $[Y = 1]$ between two groups: $[X = 0]$ and $[X = 1]$. And that would shed light on potential codependence... Feb 5 '20 at 11:19
• ... On the other hand, McNemar's test does not explore codependence of $X$ and $Y$ if they are recorded for the same subjects. Rather, McNemar tests the hypothesis of whether $P(X = 1) = P(Y = 1)$. That is a different research question, which of course is also interesting at times. Feb 5 '20 at 11:22

To me, this seems like a case where you should use Fisher's exact test.

I.e. you would make a table of your outcomes,

╔═════╦═════════════╦═════════════╗
║     ║     X=0     ║     X=1     ║
╠═════╬═════════════╬═════════════╣
║ Y=0 ║ |[X=0^Y=0]| ║ |[X=1^Y=0]| ║
║ Y=1 ║ |[X=0^Y=1]| ║ |[X=1^Y=1]| ║
╚═════╩═════════════╩═════════════╝


where e.g. |[X=0^Y=0]| are the number of data points with $$X_i$$=0 and $$Y_i$$=0.

As explained in the wikipedia entry, the values in the table will follow a hypergeometric distribution.

• Could you elaborate please? Dec 18 '19 at 17:59
• This test would apply to compare two proportions, but I don't see it working for the dependence between two binary variables.
– Dave
Jan 28 '20 at 17:12