Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or its variants towards actively managing the bias-variance trade off, or in-sample/out-of-sample surprises faced in machine learning during training and prediction?


Entropy pops up everywhere in statistical inference and machine learning. For instance:

  • Finding a parameter which maximizes the likelihood of the data is equivalent to finding a parameter which minimizes the KL divergence.
  • The Principle of Maximum Entropy is applied to find statistical models that have the greatest entropy for a set of given constraints (a way of avoiding overfitting).
  • A great example of entropy applied in machine learning is the restricted Boltzmann machine.

If you haven't already, I would check out the book Information Theory, Inference, and Algorithms by David MacKay and Probability Theory: the Logic of Science by E.T. Jaynes.

| cite | improve this answer | |
  • $\begingroup$ this a fairly general coverage of what entropy is. looking for an explanation of what it can do for surprise minimization $\endgroup$ – develarist Aug 5 at 23:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.