# Are Dispersion and Entropy Related?

So for a system, a dispersion is the measure of how the population deviates from the mean. Intuitively the more the dispersion in the system the more the disorder i.e. entropy. A jar of marbles with only red colors have 0 dispersion (if we measure dispersion by color) as well as 0 entropy. However the following scenerio confuses me in terms of dispersion and entropy:

Say a class has 2 students. Both the students obtain 10 marks in some test. The average of the class now is 10 while the variance/dispersion is 0. The entropy on the other hand is not 0 which is counter intuitive. $$\mu=\Sigma \ p(x_i)x_i=0.5(10)+0.5(10)=10$$ $$\sigma^2=\frac{\Sigma(x_i-\mu)^2}{N}=\frac{(10-10)^2+(10-10)^2}{2}=0$$ $$H(x_i)=\Sigma \ -p(x_i)\log(p(x_i))=0.5\log(2)+0.5\log(2)=1$$

• Have a look at this stats.stackexchange.com/questions/483535/… where it is shown that variance can be seen as a kind of entropy! How useful that is, is another question. Aug 27, 2021 at 14:48

You are trying to calculate entropy from the sample (just as your mean and variance are sample mean and variance as well). However, it's a function of the distribution, in this case the PMF. Let $$X$$ be the RV that denotes the grade a random student gets. You need the PMF of $$X$$ to calculate the actual entropy. But, you can still estimate it using the empirical PMF, which is: $$\hat p_X(x)=\begin{cases}1 &, x=10\\0 &,\text{else}\end{cases}$$ And the empirical entropy would be: $$\hat H (p)=1\times\log 1=0$$
• It'd check out as well when you do $1/3\times10+1/3\times10+1/3\times10=10$, but would it be correct? Define your RV, e.g. X = grade of student. And then, define your distribution. I don't see two different values of $X$ with probabilities 1/2. Apr 30, 2020 at 14:21