0
$\begingroup$

Suppose I have a response $Y$ and two features, $X$ and $Z$. Individually the features are not very predictive but their interaction is strongly predictive. Something like

$$Y = 0.5X + 0.5Z + 20XZ + \epsilon$$

Will just $X$ and $Z$ still have high mutual information with $Y$ because of the interaction term or will only the dependency of $Y$ on $X$ and $Z$ separately be captured?

$\endgroup$
1
$\begingroup$

In general, high (relatively speaking) mutual information between Y and (X,Z) does not imply high mutual information between Y and X or Y and Z.

Let X and Z be independent random variables. Let Y=XZ. Then the mutual information between Y and (X,Z) is ent(X,Z), but the mutual information between Y and X is 0.

OTOH, high mutual information between Y and X does imply high mutual information between Y and (X,Z).

I(Y,(X,Z)) >= I(Y,X)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.