Are Dispersion and Entropy Related?

Question

So for a system, a dispersion is the measure of how the population deviates from the mean. Intuitively the more the dispersion in the system the more the disorder i.e. entropy. A jar of marbles with only red colors have 0 dispersion (if we measure dispersion by color) as well as 0 entropy. However the following scenerio confuses me in terms of dispersion and entropy:

Say a class has 2 students. Both the students obtain 10 marks in some test. The average of the class now is 10 while the variance/dispersion is 0. The entropy on the other hand is not 0 which is counter intuitive. $$\mu=\Sigma \ p(x_i)x_i=0.5(10)+0.5(10)=10$$ $$\sigma^2=\frac{\Sigma(x_i-\mu)^2}{N}=\frac{(10-10)^2+(10-10)^2}{2}=0$$ $$H(x_i)=\Sigma \ -p(x_i)\log(p(x_i))=0.5\log(2)+0.5\log(2)=1$$

Answer 1

You are trying to calculate entropy from the sample (just as your mean and variance are sample mean and variance as well). However, it's a function of the distribution, in this case the PMF. Let $X$ be the RV that denotes the grade a random student gets. You need the PMF of $X$ to calculate the actual entropy. But, you can still estimate it using the empirical PMF, which is: $$\hat p_X(x)=\begin{cases}1 &, x=10\\0 &,\text{else}\end{cases}$$ And the empirical entropy would be: $$\hat H (p)=1\times\log 1=0$$