Information theory deals with signal/noise identification, while one of its tools, entropy, measures the surprise in random probabilistic outcomes. Has there been any application of using entropy or its variants towards actively managing the bias-variance trade off, or in-sample/out-of-sample surprises faced in machine learning during training and prediction?
Entropy pops up everywhere in statistical inference and machine learning. For instance:
- Finding a parameter which maximizes the likelihood of the data is equivalent to finding a parameter which minimizes the KL divergence.
- The Principle of Maximum Entropy is applied to find statistical models that have the greatest entropy for a set of given constraints (a way of avoiding overfitting).
- A great example of entropy applied in machine learning is the restricted Boltzmann machine.