2x2 chi-square test vs. binomial proportion statistic

Question

Suppose I'm doing binary classification, and I want to test whether using feature X is significant or not. (For example, I could be building a decision tree, and I want to see whether I should prune feature X or not.)

I believe the standard method is to use a chi-square test on the 2x2 table

             X = 0    X = 1
Outcome = 0      A        B
Outcome = 1      C        D

A "simpler" (IMO) test, though, would be to calculate a statistic on the probability that X gives the correct outcome: take p = [(x = 0 and Outcome = 0) + (x = 1 and Outcome = 1)] / [Total number of observations], and calculate the significance that p is far from 0.5 (say, by using a normal approximation or a Wilson score).

What are the disadvantages/advantages of this approach, compared to the chi-square method? Is it totally misguided? Are they equivalent?

Answer 1

Suppose you have

Case 1:

A=200, B=100

C=100, D=200

versus

Case 2:

A=200, B=0

C=200, D=200

The B=0 in case 2 means that case 2 provides much stronger evidence than case 1 of a relationship between X and Outcome; but in your test, both cases would be scored the same.

The Chi-Square test, informally speaking, not only takes into account the "X XOR Outcome" relationship (which is what you test) but also "X implies Outcome", "not X implies Outcome" and so on.

Answer 2

Shouldn't you favour Fisher's exact test on a 2x2 contingency table? The advantages of the Chi-squared test would be preserved, with the additional advantages of an exact test. Based on a few handbook readings, I believe that Fisher's exact test is recommended chiefly with low cell counts, but it is also often also recommended for 2x2 tables.

Answer 3

Fisher's exact was once only recommended for low cell counts because back in "the dark times" it was computationally infeasible to use it for large counts. Indeed, with some approximations doing small counts properly demands a correction be applied.

No matter, the hypergeometric tests involved are good and powerful. They can be generalized to NxM tables by deposing these into 2x2 tables, where a count of significance for row and column is kept in row 1 and in column 1, and their complements summed to form row 2 and column 2, respectively.

See http://www.stat.psu.edu/online/courses/stat504/03_2way/30_2way_exact.htm for a nice overview and the text(s) by Bishop and Fienberg as well as Agresti for a detailed presentation.

Realizing the connection with the hypergeometric lets you pick two aspects of whether an intersection represents independence or not, letting both false alarms and effect size be specified.

I like the Bishop and Fienberg, and Fienberg books:

http://www.amazon.com/Discrete-Multivariate-Analysis-Theory-Practice/dp/0387728058/

http://www.amazon.com/Analysis-Cross-Classified-Categorical-Data/dp/0387728244/

as well as the related one by Zelterman:

http://www.amazon.com/Models-Discrete-Data-Daniel-Zelterman/dp/0198567014/

Answer 4

KL divergence is another good test statistic to use. It has very similar properties to the chi-square for large expected cell counts, but retains these "good" properties even when expected cell counts are small.

The KL divergence is given by:

$$\sum_{i}O_{i}\log\left(\frac{O_{i}}{E_{i}}\right)$$

And this statistic never breaks down unless your hypothesis (the $E_{i}$ contradict the data by specifying $E_i=0$ when $O_i>0$. For independence we have $E_A=\frac{(A+B)(A+C)}{A+B+C+D}$, etc. as in the chi-square test. This is also called "likelihood ratio chi-square" in some stats packages. For the purpose of calculating a p-value, its no different to pearson chi-square (they both require large $E_i$ for the asymptotic chi-square distribution to apply). But you don't need a p-value, as this is a log likelihood ratio, so you can interpret the numerical value directly.