# Why does entropy as error measure leads to overfitting?

This post on KDnuggets quoting the post by John Langford states that entropy and mutual information when used as error measures can lead to overfitting. Could you go into more details on this?

• Note that in the original post by Langford, there is a discussion starting with Aleks' comment on how these measures are "brittle" (and can lead to overfitting). – Stephan Kolassa Feb 8 '18 at 10:55
• @StephanKolassa I noticed, but I'd be interested in more detailed comment on this. – Tim Feb 8 '18 at 10:56

In general when you fit your training data to a model which you want to generalize well to new data, this training step is accomplished by minimizing some error measure $$E (w)$$ which depends, among many things, on your parameters $$w$$ (a vector that comprises all your model parameters which are going to be fit during training).
There are techniques to avoid this, which altogether are called "regularization" techniques, being the most common the ones which add a regularization term to the error function, so that now $$E (w) = E_D (w) + E_W (w)$$ where $$E_D$$ is an error that measures how good is your fit (e.g. entropy) and $$E_W$$ a penalization for complex models (larger for models with many parameters or large parameter values).