I have frequently heard concern over "brittleness" of entropy and mutual information as performance metrics for a statistical fitting and the fact that it leads to overfitting. You can see an example of such concern in this blog post. However I have trouble understanding what exactly "brittleness" means in this context, and in which cases it would be a basis for overfitting.
- In which cases should entropy and mutual information not be used?
- If they are used, how can you ensure that no overfitting occurs?