# Can two variables' entropies be equal, as well as their joint entropy?

If two random variables $$X$$ and $$Y$$ have the same Shannon entropy, $$H(X) = H(Y)$$ can their joint entropy ever be equal to both? $$H(X,Y) = H(X) = H(Y)$$

Yes. I have an example here. (I'm still learning MathJax, so hope this is clear enough).

Consider the following 3x3 matrix that represents the joint probability from the intersection of two distributions. $$\begin{matrix} & y_i & y_j & y_k & H(X) \\ x_i& 0.333 & 0.000 & 0.000 & 0.528\\ x_j& 0.000 & 0.333 & 0.000 & 0.528\\ x_k& 0.000 & 0.000 & 0.333 & 0.528\\ H(Y) & 0.528 & 0.528 & 0.528 & 1.585\\ \end{matrix}$$

As can be seen, the marginal entropies, $$\mathit{H(X)}$$ (rows) and $$\mathit{H(Y)}$$ (cols), are both equal to 1.585. The joint entropy, $$\mathit{H(X,Y)}$$, by my calculations, equals 1.585 as well. In this case, the mutual information, $$\mathit{I(X;Y)}$$, is also equal to 1.585. This agrees with the following identity:

$$\mathit{I(X;Y)=H(X)+H(Y)-H(X,Y)}$$ $${1.585=1.585+1.585-1.585}$$

This occurs when all the information conveyed by $$\mathit{X}$$ is shared with $$\mathit{Y}$$.

• could you unravel this to the 2 data series' non-joint probabilities please so i can calculate everything as well? Commented Aug 27, 2020 at 11:58
• @develarist. I have a spreadsheet here that I use for my modelling with the full calculations but not sure how I can present them here. I will see what I can do. Commented Aug 27, 2020 at 21:48
• even a small dataset of 10 observations (probabilities) for two variables $X$ and $Y$ would do Commented Sep 7, 2020 at 2:55
• @ develarist. Sorry for the delay in replying. Consider the paired results (ordered) for the before test $\mathit{(X)}$ and after test $\mathit{(Y)}$ where $\mathit{X}$ = {1,1,1,2,2,2,3,3,3} and $\mathit{Y}$ = {1,1,1,2,2,2,3,3,3}. For this example, $\mathit{X = Y}$. The distributions are uniform and the joint probability matrix is given above where the indexes relate to $\mathit{i}$ = 1, $\mathit{j}$ = 2 and $\mathit{k}$ = 3. It follows that $\mathit{H(X) = H(Y) = H(X,Y) = I(X;Y) = 1.585}$. Commented Sep 11, 2020 at 8:35
• How about a case where $X\neq Y$? Commented Sep 11, 2020 at 11:19