Tsallis and Rényi Normalized Entropy

Question

I'm working with Shannon, Tsallis and Rényi entropies. I need to normalize these entropies for comparison purposes. In Shannon's entropy you need only to divide by the log of the number of bins.

$$H(X) = -\sum_{i}\left({P(x_i) \log_b P(x_i)}\right)/\log_b(N)$$

where $N$ is the number of bins and $b$ the log-base (in Shannon is equal 2).

Edit: Also for Rényi it is $\log(N)$

I'm missing Tsallis.

Answer 1

Tsallis and Rényi entropy is the same thing, up to some rescaling. All of them are functions of $\sum_i p_i^\alpha$, with the special case of $\alpha\to1$ giving Shannon entropy.

Look at Tom Leinster's "Entropy, Diversity and Cardinality (Part 2)", especially at the table comparing these properties.

In short:

• Rényi entropies are in $[0, \log(N)]$,
• Tsallis entropies (called there $\alpha$-diversities) are in $[0, (1-N^{1-\alpha})/(1-\alpha)]$,
• $\alpha$-cardinalities are in $[1, N]$.

Also, one more way to go is to use:

• 1/cardinality, in $[\tfrac{1}{N}, 1]$,
• just $\sum_i p_i^\alpha$, in $[\tfrac{1}{N^{\alpha-1}}, 1]$.

The later two have the advantage that no matter what is the $N$, they always end up in $[0, 1]$.

Answer 2

The answer provided by Piotr is mostly correct, but there is a tiny mistake.

The maximum Tsallis entropy value is actually given by the expression: $(1−N^{1−\alpha})/(\alpha-1)$.

The denominator order is reversed.

This is because this expression is obtained when dealing with a uniform probability distribution $ P = \{\frac{1}{N}, \frac{1}{N}, ..., \frac{1}{N}\}$

Considering the Tsallis entropy defined by the expression:

\begin{align} T &= \frac{1}{\alpha-1} \sum_{j=1}^{N}(P_{j} - (P_{j})^\alpha)\\ \end{align}

In this particular scenario, all instances of $ P_{j} $ will be equal to $\frac{1}{N}$. This means the sum can be reduced to $N\times(\frac{1}{N}-(\frac{1}{N})^\alpha)$. Therefore, we can rewrite this in the following way:

\begin{align} T_{\max} &= \frac{1}{α-1}\times\left \{N\times\left[\frac{1}{N}-\left(\frac{1}{N}\right)^\alpha\right]\right\}\\ T_{\max} &= \frac{1}{\alpha-1}\times (1-N^{1-\alpha})\\ T_{\max} &= \frac{1−N^{1−\alpha}}{\alpha-1}\\ \end{align}