This post on KDnuggets quoting the post by John Langford states that entropy and mutual information when used as error measures can lead to overfitting. Could you go into more details on this?

  • 3
    $\begingroup$ Note that in the original post by Langford, there is a discussion starting with Aleks' comment on how these measures are "brittle" (and can lead to overfitting). $\endgroup$ – Stephan Kolassa Feb 8 '18 at 10:55
  • $\begingroup$ @StephanKolassa I noticed, but I'd be interested in more detailed comment on this. $\endgroup$ – Tim Feb 8 '18 at 10:56

Any error measure which does not penalize the complexity of the system may lead to overfitting, e.g. entropy.

In general when you fit your training data to a model which you want to generalize well to new data, this training step is accomplished by minimizing some error measure $E (w) $ which depends, among many things, on your parameters $w $ (a vector that comprises all your model parameters which are going to be fit during training).

If your error measure just cares about fitting better and better your training data, you may find that building models with a huge number of parameters (that aditionally may take any value) is good because your model is so flexible that your training data can be perfectly learnt. On the other hand, if your training data is noisy (which is usually the case) you will this way make your model fit noise also, and this is what overfitting is about.

There are techniques to avoid this, which altogether are called "regularization" techniques, being the most common the ones which add a regularization term to the error function, so that now $E (w) = E_D (w) + E_W (w) $ where $E_D$ is an error that measures how good is your fit (e.g. entropy) and $E_W$ a penalization for complex models (larger for models with many parameters or large parameter values).

| cite | improve this answer | |
  • 3
    $\begingroup$ This does not seem to answer the question. John Langford mentions entropy explicitly as a measure that leads to overfitting, he does not mention "any measure". The question is, what/if is specific about entropy in terms of overfitting? $\endgroup$ – Tim Mar 6 '19 at 7:11
  • $\begingroup$ regularization is not generally considered on validation/test sets, for all measures. It seems that Langford was talking about entropy and mutual information in particular. $\endgroup$ – carlo May 8 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.