# When and why does the “brittleness” of mutual information cause overfitting?

I have frequently heard concern over "brittleness" of entropy and mutual information as performance metrics for a statistical fitting and the fact that it leads to overfitting. You can see an example of such concern in this blog post. However I have trouble understanding what exactly "brittleness" means in this context, and in which cases it would be a basis for overfitting.

• In which cases should entropy and mutual information not be used?
• If they are used, how can you ensure that no overfitting occurs?

• In general, mutual information needs some data (often more than other methods). If you see counts 1 or 2 appearing often, it means that mutual information will overfit. The only way to deal with it is to bin some data (e.g. if you have values of one variable 1,2,3 and 4 then you can use instead [1,2] and [3,4]). – Piotr Migdal Mar 31 '14 at 11:17