2
$\begingroup$

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further rearrangement produces the following:

$-\sum_{i=1}^k\frac{1}{k}\log(k)^{-1}$

$\sum_{i=1}^k\frac{1}{k}\log(k)$

This is where I am stuck. I need the solution to come to $\log(k)$. What is the next step?

$\endgroup$
  • 1
    $\begingroup$ You have the sum of $(1/k) \log k$, each repeated $k$ times. Try this one. What is $1/k$ repeated $k$ times? Just $k (1/k) = 1$. $\endgroup$ – Nick Cox Apr 20 '19 at 9:34
1
$\begingroup$

This is a constrained maximization problem in $k$ variables $p_1,p_2,...p_k$. The objective function is

$$-\sum_{i=1}^kp_i\log(p_i)$$

and the constraint is

$$\sum_{i=1}^kp_i = 1$$

Form the Lagrangean and I guess you can proceed from here.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Using Lagrange multipliers we have the equation:

$$\mathcal{L} = \left \{ -\sum_i^k p_i \log p_i - \lambda\left ( \sum_i^k p_i - 1 \right )\right \}$$

Maximizing with respect to the probability,

$$\frac{\partial \mathcal{L}}{\partial p_i} = 0 = -\log p_i - 1 - \lambda \implies $$

$$p_i = e^{-(1+\lambda)}\tag{1}$$

Maximizing with respect to $\lambda$:

$$\frac{\partial \mathcal{L}}{\partial \lambda} = 0 = - \sum_i^k p_i + 1 \implies$$

$$ \sum_i^k p_i = 1 \tag{2}$$

Substituting equation (1) into equation (2):

$$\sum_i^k e^{-(1+\lambda)} = 1 \implies$$

$$k e^{-(1+\lambda)} = 1 $$

Since $p_i = e^{-(1+\lambda)}$

$$p_i = \frac{1}{k}$$

The Shannon Entropy formula now becomes

$$ H = - \sum_i^k \frac{1}{k}\log \frac{1}{k}$$

Since $k$ does not depend on the summation,

$$H = \frac{k}{k} \log k = \log k$$

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.