Testing if there is any new knowledge in a third variable

Question

I am trying to predict the outcome $X$ of an event given a big set of input variables $A$, $B$, $C$, $D$, etc.

Both $X$ and $A$, $B$, $C$, etc. are categorical variables, some of them with a high number of categories.

We can think of $A$ as the main input because there is a hard correlation between $X$ and $A$. Also, most of the other variables $B$, $C$, $D$, etc. have a high correlation with $A$ too.

In order to reduce the size of my input vector I was thinking about some way to remove those variables that don't increase my knowledge about $X$ farther once I know $A$.

I am considering performing a $\chi^2$ test for every triplet $(X, A, anyOtherVar)$. My reasoning is that $p_{(A,B,X)} = p_A*p_{(B,X|A)}$ and if we assume that $B$ does not increase my knowledge about $X$ once I know $A$, then $p_{(A,B,X)} = p_A*p_{(B|A)}*p_{(X|A)}$ which becomes my null hypothesis.

And so, I can build the 3-dimensional contingency table using the probability from my null hypothesis $p_A*p_{(B|A)}*p_{(X|A)}$ instead of the usual $p_A*p_B*p_X$.

Does that procedure looks sound to you? Are there any other known and better approaches to attain my objective?

Update: Because of the hard correlations, I have a lot of zeros in the $p_{B|A}$ and $p_{X|A}$ matrices which result in divisions by zero in the $\chi^2$ addends. I can remove the cells with 0 predicted elements from the table, but then I am finding quite difficult to come with a method to calculate the degrees of freedom.

Answer 1

A reasonable approach for your problem is forward selection, a type of Stepwise Regression.

You could fit a regression with just the $A$ variables, and then compare this to the regression with the $A,B$, $A,C$, and $A,D$ variables via F-test where, for example

$$F_{A \; \text{vs.} \, A,B}=\frac{(SSE_A-SSE_{A,B})/(df_{A}-df_{A,B})}{SSE_{A,B}/df_{A,B}}.$$

Doing this for $A,B$, $A,C$, and $A,D$, you then add the variable to the model which had the highest value in the $F$ test (or if no variable was significant in the $F$ test you are done and can just stick with using $A$). You can then continue on in this manner to see if adding a third variable makes sense.

You can also try backward selection to see if it gives similar results.

Answer 2

The best method to achieve this would be a PCA or factor analysis on the variables before you do the regression. Think about it this way: If $A$ and $X$ are highly correlated, and $B$ and $X$ are also highly correlated, then very likely (but not necessarily) $A$ and $B$ are also correlated. So your variables $A,B,C,\ldots$ don't really capture independent information. This leads to problems in the analysis. So it is better to first remove this correlation and identify the underlying actual information in the variables. A PCA can achieve this.

Answer 3

It is not true that $\Pr(X|A, B) = \Pr(X|A)$ implies $\Pr(X|A, B, C) = \Pr(X|A, C)$: Given, $A$, a variable might predict $X$ together with another variable $C$ but not on it's own. And the same could hold for $C$.

Here's a stark example illustrating the point. Let $B$ be a coinflip. $C$ is another, independent, one. $X$ is a light. It turns on if $B$ and $C$ take the same value. $B$ is clearly useless for predicting $X$. As is $C$. But if you know both, you can perfectly predict $X$.

Considering one variable at a time in addition to $A$ would fail to uncover such interactions. It might be too aggressive and remove variables that would help.

But it might also be too conservative and fail to remove variables that are useless. If one variable is a copy of another, they'd both be kept even as only one of them is useful. From what you describe, this is likely in your case (the variables are highly correlated)

An alternative approach I'd suggest would be to estimate the additional predictive powers of large sets of variables in addition to $A$ jointly. Consider using a model with Lasso or Elastic Net penalty.

Answer 4

As @David Veich already mentioned a good way to proceed further would be feature selection methods like Forward selection,Backward elimination or Recursive feature elimination. In place of these embedded Elimination methods you can also apply a Lasso Regression model. You can also look into discriminant correspondence analysis which is an extension of Linear Discriminant Analysis for Categorical Variables.

Do view this links for more information: Lasso Regression for Categorical Variables

Discriminant Correspondence Analysis