I am trying to calculate Mutual Information scores for Feature Selection.

I have successfully implemented the Mutual Information to test each feature against the binary response variable. Each feature in my case is an n-gram with values 1(appears), 0(does not appear), and the binary response class takes values 0 or 1.

However, I now want to apply the same test, but this time the response variable takes three values(0,1,2).

I have searched for a similar example but I haven't found anything online. Is this possible to perform?

If yes, is this a simple expansion to the existing formula, or does this introduce a more complicated problem?

  • $\begingroup$ sure mutual information formula can be computed for response variables with $k$ values $\endgroup$
    – Simone
    Jul 21, 2014 at 1:08

1 Answer 1


The formula for mutual information does not depend on the number of values that the random variables can take. Simply sum over all possible values, i.e. 0, 1, and 2 in your case.

$ I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) } $


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.