New answers tagged

0

For me, the best intuition (or even derivation for why the forward KL is useful) comes from information theory (optimal codes specifically). I'll introduce the basics below; for much more detail, please see Chapter 5 of "Elements of Information Theory" (Thomas & Cover). Suppose you have a message $x$ which we assume is a realisation of a RV X; ...


0

Variance is not similar to entropy at all. The variance has a scale, and entropy doesn't. Yes, the variance measures the uncertainty, but in a very different manner compared to the entropy. The variance, and especially its cousin standard deviation, reflect the absolute uncertainty. For instance, you could say that the volatility of S&P 500 index annual ...


0

There are other measures of central tendency besides the mean. The median and mode are also measures of central tendency. Unlike the mean and median, the mode can be generalized to information theory: the value in the domain with the highest probability or probability density. So information theory does have an analog to central tendency, even if it does not ...


3

The amount of potential information contained in a signal is what we call the entropy, usually denoted by $H$ and defined as follows: $$H(X) = − \sum_X{P(X) \log P(X)}$$ The "surprise" is (or you could say proportional to, but it doesn't matter) the inverse of the probability and you can see that the example you mentioned. So the entropy can be ...


Top 50 recent answers are included