# How to include the observed values, not just their probabilities, in information entropy?

Shannon entropy measures the unpredictability in a random variable's outcome as the weighted average of the probabilities of that variable's outcomes or observed values. However, it discards the actual observed values that the probabilities were derived from, only using probabilities in its formula instead. This seems like a significant information loss since it is the observed values, not their probabilities, that contain details like magnitude and direction of the random variable's realizations.

Aren't there any estimators of entropy that include the observed values alongside the probabilities, so that it is not being measured based on probabilities alone? For example, an adjusted entropy measure that is a weighted average of the probabilities and observed values somehow?

Entropy can be developed as expected surprise, as I tried in Statistical interpretation of Maximum Entropy Distribution. I will work for the discrete case now, but most can be carried over to the continuous case.

Define a surprise function $$\sigma \colon p \mapsto \sigma(p)$$ which sends a probability to the surprise value of an event having that probability. As you gets less surprised by a more probable event, $$\sigma$$ should be decreasing, and $$\sigma(1)=0$$ as you are not surprised at all by a certain event occurring. The surprise function $$\log\left( \frac1p \right)$$ leads to Shannon entropy.

First, lets connect this to the example in the other answer by @Richard Hardy. Denote the discrete values by $$x_i$$ and suppose they are real numbers. Suppose there is one $$x_m =\mu$$, the expected value, and that $$p(x_i)\leq p(x_m)$$, and if $$(x_i-\mu)^2 \geq (x_j-\mu)^2$$ then so $$p_i \leq p_j$$. In this case $$\sigma \colon x\mapsto (x-\mu)^2$$ is a surprise function and the corresponding expected surprise is the variance. This way we can consider the variance a kind of entropy!

Back to generalities. A family of surprise functions is $$\sigma_\alpha \colon [0,1]\mapsto [0, \infty]; \quad \sigma_\alpha(p)=\begin{cases} (1-\alpha)^{-1} (1-p^{\alpha-1})&, \alpha\not=1 \\ -\log p &, \alpha=1 \end{cases}$$ The expected surprise becomes $$D_\alpha(p_1, \dotsc, p_n)=\sum_i p_i \sigma_\alpha(p_i) = \\ \begin{cases} (\alpha-1)^{-1} (\left( 1-\sum_i p_i^\alpha\right) &, \alpha\not=1 \\ -\sum_i p_i\log p_i &, \alpha=1 \end{cases}$$ and we have used the name $$D$$ because in ecology this is known as diversity (as in biodiversity.) In ecology one often presents this in another way using the concept of effective number of species. The idea is that an ecosystem with $$n$$ species is most diverse if the frequency of all the species are the same, so $$p_i=1/n$$. In other cases we can calculate some $$\text{effective number of species }\leq n$$. I wrote about that here: How is the Herfindahl-Hirschman index different from entropy? so will not repeat. In the case of the Shannon entropy the effective number of species is given by the exponential of the entropy. Now write $$A=\{p_1, \dotsc, p_n\}$$ and $$\lvert A \rvert = e^{H(A)} =\prod_i p_i^{-p_i}$$ and call this the cardinality of $$A$$, to have a mathematical name useful also outside ecology. Think of this as a measure of the size of $$A$$. Now we want to extend this for all the surprise functions $$\sigma_\alpha$$. The result is (for the moment I jump the development) $$\lvert A \rvert_\alpha = \begin{cases} \left( \sum_i p_i^\alpha\right)^{\frac1{1-\alpha}}&,\alpha\not=1 \\ \prod_i p_i^{-p_i}&, \alpha=1 \end{cases}$$ Now we can get back to the entropy scale by taking logarithms, and so we define the $$\alpha$$-entropy by $$H_\alpha(A)=\log \lvert A \rvert_\alpha$$. This is usually called the Renyi-entropy, and has better mathematical properties than the $$\alpha$$-diversity. All of this and more can be found starting here.

The measures we have discussed so far only use the probabilities $$p_i$$, so we did not answer the question yet---so some patience! First we need a new concept:

Cardinality of metric spaces Let $$A$$ be a set of points $$a_1, \dotsc, a_n$$ with given distances $$d_{ij}$$ ($$d_{ij}=\infty$$ is permitted.) Think of this as a finite metric space, but it is not clear we really need all the metric space axioms. Define a matrix $$Z=\left( e^{-d_{ij}}\right)_{i,j}$$ and a vector $$w$$ as any solution of $$Z w = \left(\begin{smallmatrix}1\\ \vdots \\1 \end{smallmatrix}\right)$$. $$w$$ is called a weighting of $$A$$. Now we can define the cardinality of $$A$$ as the sum of the components of $$w$$, $$\lvert A \rvert_\text{MS} =\sum_i w_i$$ It is an exercise to show this does not depend on the choice of $$w$$. Now we want to extend this definition to a ...

Metric probability space $$A=(p_1, \dotsc, p_n; d)$$ where $$d$$ is a distance function, a metric. To each point $$i$$ we associate a density $$\sum_j p_j e^{-d_{ij}}$$. Since $$e^{-d_{ij}}$$ is antimonotone in the distance $$d$$, it represents a closeness, so the density can be seen as an expected closeness around point $$i$$, which explains the terminology density. Define a similarity matrix $$Z=\left( e^{-d_{ij}}\right)_{i,j}$$ and probability vector $$p=(p_1, \dotsc, p_n)$$. Now $$Zp$$ is the density vector. For example, if all distances $$d_{ij}=\infty$$ then $$Z=I$$, the identity matrix, so $$Zp=p$$.

Now we will generalize by replacing in many formulas $$p$$ with $$Zp$$.

Earlier surprise only depended upon the probabilities of the observed event. Now we will also take into account the probabilities of nearby points. For instance , you will probably be very surprised by a python snake at Manhattan, but now we will measure that surprise also taking into account the probabilities of other snakes ... with the surprise function $$\sigma$$, the expected surprise is now defined as $$\sum_i p_i \sigma\left( (Zp)_i\right)$$ for a discrete metric space with all $$d_{ij}=\infty$$, this is no change.

Diversity is now generalized to $$D_\alpha(A)=\sum_i p_i \sigma_\alpha\left( (Zp)_i\right)= \begin{cases} (\alpha-1)^{-1} \left(1-\sum_i p_i(Zp)_i^{\alpha-1} \right)&,\alpha\not=1 \\ -\sum_i p_i \log\left( (Zp)_i\right) &, \alpha=1\end{cases}$$ For example, with $$\alpha=2$$, $$D_2(A)= p^T \Delta p$$, $$\Delta=\left( 1-e^{-d_{ij}}\right)_{i,j}$$ is known as Rao's quadratic diversity index, or Rao's quadratic entropy.

$$\alpha$$-Cardinality Correspondingly we have $$\lvert A\rvert_{\alpha} = \frac1{\sigma_\alpha^{-1}(D_\alpha(A))}= \begin{cases} \left( \sum_i p_i (Zp)_i^{\alpha-1} \right)^{\frac1{1-\alpha}}&,\alpha\not=1 \\ \prod_i (Zp)_i^{-p_i} &, \alpha=1 \end{cases}$$ and now the ...

$$\alpha$$-entropy is obtained by taking the logarithms of the $$\alpha$$-cardinality, and this way we now have obtained an entropy where distances between the points plays a role. All this and much more can be found here at the n-Category cafe. This is still relatively new theory, so new developments can be expected. The ideas comes originally from theoretical ecologists.

Entropy measures the amount of randomness or surprise of a random phenomenon/experiment, not necessarily a random variable (the latter need not even be defined).

Regarding your question, measures of spread such as mean absolute deviation, variance and the like could be relevant. E.g. variance could indeed be considered an adjusted entropy measure that is a weighted average of the probabilities and [observed] values. For a continuous random variable with expectation $$\mu_X$$ and probability density $$f(x)$$, $$\text{Var}(X)=\int_{-\infty}^{\infty}(x-\mu_X)^2f(x)\ dx;$$ for a discrete one with possible values $$x_1,\dots,x_n$$ with the corresponding probabilities $$p_1,\dots,p_n$$ and with expectation $$\mu_X$$, it is $$\text{Var}(X)=\sum_{i=1}^{n}(x_i-\mu_X)^2 p_i.$$ You can see both the possible values and their probabilities/densities playing a role.

• but variance measures dispersion by only using observed values. It too is only looking at one side of the data like how I explained for entropy, which uses only probabilities. the question asks how to incorporate both observed values and their probabilities Aug 18 '20 at 12:00
• @develarist, variance looks at probabilities, too. Check out the formula of variance for a continuous random variable, you will see integration w.r.t. to the pdf. For a discrete random variable, there will be summation w.r.t. the pmf. Aug 18 '20 at 12:01
• so you're saying variance does take into consideration both observed values and probabilities whereas entropy only looks at probabilities. Does that make variance superior then Aug 18 '20 at 12:02
• @develarist, we need a criterion to define what superior means. Depends on the purpose. Aug 18 '20 at 12:05
• @develarist, see "What is the role of the logarithm in Shannon's entropy?". As for variance, it is a different animal and there is no logical need for a logarithm there. Aug 18 '20 at 14:59