# Proving that Shannon entropy is maximised for the uniform distribution

I know that Shannon entropy is defined as $$-\sum_{i=1}^kp_i\log(p_i)$$. For the uniform distribution, $$p_i=\frac{1}{k}$$, so this becomes $$-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$$. Further rearrangement produces the following:

$$-\sum_{i=1}^k\frac{1}{k}\log(k)^{-1}$$

$$\sum_{i=1}^k\frac{1}{k}\log(k)$$

This is where I am stuck. I need the solution to come to $$\log(k)$$. What is the next step?

• You have the sum of $(1/k) \log k$, each repeated $k$ times. Try this one. What is $1/k$ repeated $k$ times? Just $k (1/k) = 1$. – Nick Cox Apr 20 '19 at 9:34

## 2 Answers

This is a constrained maximization problem in $$k$$ variables $$p_1,p_2,...p_k$$. The objective function is

$$-\sum_{i=1}^kp_i\log(p_i)$$

and the constraint is

$$\sum_{i=1}^kp_i = 1$$

Form the Lagrangean and I guess you can proceed from here.

Using Lagrange multipliers we have the equation:

$$\mathcal{L} = \left \{ -\sum_i^k p_i \log p_i - \lambda\left ( \sum_i^k p_i - 1 \right )\right \}$$

Maximizing with respect to the probability,

$$\frac{\partial \mathcal{L}}{\partial p_i} = 0 = -\log p_i - 1 - \lambda \implies$$

$$p_i = e^{-(1+\lambda)}\tag{1}$$

Maximizing with respect to $$\lambda$$:

$$\frac{\partial \mathcal{L}}{\partial \lambda} = 0 = - \sum_i^k p_i + 1 \implies$$

$$\sum_i^k p_i = 1 \tag{2}$$

Substituting equation (1) into equation (2):

$$\sum_i^k e^{-(1+\lambda)} = 1 \implies$$

$$k e^{-(1+\lambda)} = 1$$

Since $$p_i = e^{-(1+\lambda)}$$

$$p_i = \frac{1}{k}$$

The Shannon Entropy formula now becomes

$$H = - \sum_i^k \frac{1}{k}\log \frac{1}{k}$$

Since $$k$$ does not depend on the summation,

$$H = \frac{k}{k} \log k = \log k$$