3
$\begingroup$

So for a system, a dispersion is the measure of how the population deviates from the mean. Intuitively the more the dispersion in the system the more the disorder i.e. entropy. A jar of marbles with only red colors have 0 dispersion (if we measure dispersion by color) as well as 0 entropy. However the following scenerio confuses me in terms of dispersion and entropy:

Say a class has 2 students. Both the students obtain 10 marks in some test. The average of the class now is 10 while the variance/dispersion is 0. The entropy on the other hand is not 0 which is counter intuitive. $$\mu=\Sigma \ p(x_i)x_i=0.5(10)+0.5(10)=10$$ $$\sigma^2=\frac{\Sigma(x_i-\mu)^2}{N}=\frac{(10-10)^2+(10-10)^2}{2}=0$$ $$H(x_i)=\Sigma \ -p(x_i)\log(p(x_i))=0.5\log(2)+0.5\log(2)=1$$

$\endgroup$
1
1
$\begingroup$

You are trying to calculate entropy from the sample (just as your mean and variance are sample mean and variance as well). However, it's a function of the distribution, in this case the PMF. Let $X$ be the RV that denotes the grade a random student gets. You need the PMF of $X$ to calculate the actual entropy. But, you can still estimate it using the empirical PMF, which is: $$\hat p_X(x)=\begin{cases}1 &, x=10\\0 &,\text{else}\end{cases}$$ And the empirical entropy would be: $$\hat H (p)=1\times\log 1=0$$

$\endgroup$
9
  • $\begingroup$ I quite didnt understand it. Why my computation of entropy for the sample is wrong? Is it possible to elaborate the answer a bit. $\endgroup$ Apr 30 '20 at 14:11
  • $\begingroup$ Sure, it's not correct because you're using the correct probability mass function. What is your distribution of grades in the classroom? You're saying that the grade is 10 with 1/2 prob, and the grade is again 10 with again 1/2 prob. $\endgroup$
    – gunes
    Apr 30 '20 at 14:14
  • $\begingroup$ But I used the same 1/2 prob for both in the mean formula and it checks out. Right? $\endgroup$ Apr 30 '20 at 14:20
  • 1
    $\begingroup$ It'd check out as well when you do $1/3\times10+1/3\times10+1/3\times10=10$, but would it be correct? Define your RV, e.g. X = grade of student. And then, define your distribution. I don't see two different values of $X$ with probabilities 1/2. $\endgroup$
    – gunes
    Apr 30 '20 at 14:21
  • $\begingroup$ I see. So my mean formula is wrong? $\endgroup$ Apr 30 '20 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.