Skip to main content

A mathematical quantity designed to measure the amount of randomness of a random variable.

Entropy is a mathematical quantity designed to quantify the uncertainty about the occurrence of outcomes of a random variable. It is expressed as a function of the outcome probabilities of the random variable. Any measure for entropy must satisfy a few conditions:

  1. Continuity: The function must be continuous in all its arguments.
  2. Maximum: The function should have a maximum when all outcome are equally probable.
  3. Symmetry: The function must remain unchanged under a switch of arguments.

A commonly adopted measure is the Shannon entropy, $\mathrm{H}(p_1,p_2,\ldots,p_n)$ (when $p_1,p_2,\ldots p_n$ are the $n$ outcome probabilities of a random variable $X$). This measure is defined as follows:

$$\mathrm{H}(p_1,p_2,\ldots,p_n) = -\sum_{i=1}^n p_i \log p_i$$