Feature Selection - Mutual Information with response variable that takes three values

Question

I am trying to calculate Mutual Information scores for Feature Selection.

I have successfully implemented the Mutual Information to test each feature against the binary response variable. Each feature in my case is an n-gram with values 1(appears), 0(does not appear), and the binary response class takes values 0 or 1.

However, I now want to apply the same test, but this time the response variable takes three values(0,1,2).

I have searched for a similar example but I haven't found anything online. Is this possible to perform?

If yes, is this a simple expansion to the existing formula, or does this introduce a more complicated problem?

Answer 1

The formula for mutual information does not depend on the number of values that the random variables can take. Simply sum over all possible values, i.e. 0, 1, and 2 in your case.

$ I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) } $