I have frequently heard concern over "brittleness" of entropy and mutual information as performance metrics for a statistical fitting and the fact that it leads to overfitting. You can see an example of such concern in this blog post. However I have trouble understanding what exactly "brittleness" means in this context, and in which cases it would be a basis for overfitting.

  • In which cases should entropy and mutual information not be used?
  • If they are used, how can you ensure that no overfitting occurs?

I am not sure what does "brittle" mean in statistics, but in case it looks like "Traditional overfitting" - with too few data points it is easy to max mutual information, even though there is no dependence between variables.

  • $\begingroup$ Thanks for the answer, Piotr. Just to make sure: you mean that in case we have too few datapoints with a lot of features, it is easy to find identical features for those of them and attribute a maximal informativeness to those features, even if in fact they are completely random. What would be a good way of avoiding such overfitting or at least verifying for it? $\endgroup$ – Andrei Kucharavy Mar 31 '14 at 3:46
  • 1
    $\begingroup$ In general, mutual information needs some data (often more than other methods). If you see counts 1 or 2 appearing often, it means that mutual information will overfit. The only way to deal with it is to bin some data (e.g. if you have values of one variable 1,2,3 and 4 then you can use instead [1,2] and [3,4]). $\endgroup$ – Piotr Migdal Mar 31 '14 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.