0
$\begingroup$

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further rearrangement produces the following:

$-\sum_{i=1}^k\frac{1}{k}\log(k)^{-1}$

$\sum_{i=1}^k\frac{1}{k}\log(k)$

This is where I am stuck. I need the solution to come to $\log(k)$. What is the next step?

$\endgroup$
  • 1
    $\begingroup$ You have the sum of $(1/k) \log k$, each repeated $k$ times. Try this one. What is $1/k$ repeated $k$ times? Just $k (1/k) = 1$. $\endgroup$ – Nick Cox Apr 20 '19 at 9:34
1
$\begingroup$

This is a constrained maximization problem in $k$ variables $p_1,p_2,...p_k$. The objective function is

$$-\sum_{i=1}^kp_i\log(p_i)$$

and the constraint is

$$\sum_{i=1}^kp_i = 1$$

Form the Lagrangean and I guess you can proceed from here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.