3
$\begingroup$

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further rearrangement produces the following:

$-\sum_{i=1}^k\frac{1}{k}\log(k)^{-1}$

$\sum_{i=1}^k\frac{1}{k}\log(k)$

This is where I am stuck. I need the solution to come to $\log(k)$. What is the next step?

$\endgroup$
1
  • 1
    $\begingroup$ You have the sum of $(1/k) \log k$, each repeated $k$ times. Try this one. What is $1/k$ repeated $k$ times? Just $k (1/k) = 1$. $\endgroup$
    – Nick Cox
    Apr 20, 2019 at 9:34

2 Answers 2

7
$\begingroup$

Using Lagrange multipliers we have the equation:

$$\mathcal{L} = \left \{ -\sum_i^k p_i \log p_i - \lambda\left ( \sum_i^k p_i - 1 \right )\right \}$$

Maximizing with respect to the probability,

$$\frac{\partial \mathcal{L}}{\partial p_i} = 0 = -\log p_i - 1 - \lambda \implies $$

$$p_i = e^{-(1+\lambda)}\tag{1}$$

Maximizing with respect to $\lambda$:

$$\frac{\partial \mathcal{L}}{\partial \lambda} = 0 = - \sum_i^k p_i + 1 \implies$$

$$ \sum_i^k p_i = 1 \tag{2}$$

Substituting equation (1) into equation (2):

$$\sum_i^k e^{-(1+\lambda)} = 1 \implies$$

$$k e^{-(1+\lambda)} = 1 $$

Since $p_i = e^{-(1+\lambda)}$

$$p_i = \frac{1}{k}$$

The Shannon Entropy formula now becomes

$$ H = - \sum_i^k \frac{1}{k}\log \frac{1}{k}$$

Since $k$ does not depend on the summation,

$$H = \frac{k}{k} \log k = \log k$$

$\endgroup$
3
$\begingroup$

This is a constrained maximization problem in $k$ variables $p_1,p_2,...p_k$. The objective function is

$$-\sum_{i=1}^kp_i\log(p_i)$$

and the constraint is

$$\sum_{i=1}^kp_i = 1$$

Form the Lagrangean and I guess you can proceed from here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.