Can entropy be used to minimize prediction surprises in machine learning?

Question

Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or its variants towards actively managing the bias-variance trade off, or in-sample/out-of-sample surprises faced in machine learning during training and prediction?

Answer 1

Entropy pops up everywhere in statistical inference and machine learning. For instance:

• Finding a parameter which maximizes the likelihood of the data is equivalent to finding a parameter which minimizes the KL divergence.
• The Principle of Maximum Entropy is applied to find statistical models that have the greatest entropy for a set of given constraints (a way of avoiding overfitting).
• A great example of entropy applied in machine learning is the restricted Boltzmann machine.

If you haven't already, I would check out the book Information Theory, Inference, and Algorithms by David MacKay and Probability Theory: the Logic of Science by E.T. Jaynes.