Suppose I'm doing binary classification, and I want to test whether using feature X is significant or not. (For example, I could be building a decision tree, and I want to see whether I should prune feature X or not.)
I believe the standard method is to use a chi-square test on the 2x2 table
X = 0 X = 1 Outcome = 0 A B Outcome = 1 C D
A "simpler" (IMO) test, though, would be to calculate a statistic on the probability that X gives the correct outcome: take p = [(x = 0 and Outcome = 0) + (x = 1 and Outcome = 1)] / [Total number of observations], and calculate the significance that p is far from 0.5 (say, by using a normal approximation or a Wilson score).
What are the disadvantages/advantages of this approach, compared to the chi-square method? Is it totally misguided? Are they equivalent?