1
$\begingroup$

If two random variables $X$ and $Y$ have the same Shannon entropy, $$H(X) = H(Y)$$ can their joint entropy ever be equal to both? $$H(X,Y) = H(X) = H(Y)$$

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes. I have an example here. (I'm still learning MathJax, so hope this is clear enough).

Consider the following 3x3 matrix that represents the joint probability from the intersection of two distributions. $$ \begin{matrix} & y_i & y_j & y_k & H(X) \\ x_i& 0.333 & 0.000 & 0.000 & 0.528\\ x_j& 0.000 & 0.333 & 0.000 & 0.528\\ x_k& 0.000 & 0.000 & 0.333 & 0.528\\ H(Y) & 0.528 & 0.528 & 0.528 & 1.585\\ \end{matrix} $$

As can be seen, the marginal entropies, $\mathit{H(X)}$ (rows) and $\mathit{H(Y)}$ (cols), are both equal to 1.585. The joint entropy, $\mathit{H(X,Y)}$, by my calculations, equals 1.585 as well. In this case, the mutual information, $\mathit{I(X;Y)}$, is also equal to 1.585. This agrees with the following identity:

$$\mathit{I(X;Y)=H(X)+H(Y)-H(X,Y)}$$ $${1.585=1.585+1.585-1.585}$$

This occurs when all the information conveyed by $\mathit{X}$ is shared with $\mathit{Y}$.

$\endgroup$
10
  • $\begingroup$ could you unravel this to the 2 data series' non-joint probabilities please so i can calculate everything as well? $\endgroup$
    – develarist
    Commented Aug 27, 2020 at 11:58
  • $\begingroup$ @develarist. I have a spreadsheet here that I use for my modelling with the full calculations but not sure how I can present them here. I will see what I can do. $\endgroup$
    – Mari153
    Commented Aug 27, 2020 at 21:48
  • $\begingroup$ even a small dataset of 10 observations (probabilities) for two variables $X$ and $Y$ would do $\endgroup$
    – develarist
    Commented Sep 7, 2020 at 2:55
  • $\begingroup$ @ develarist. Sorry for the delay in replying. Consider the paired results (ordered) for the before test $\mathit{(X)}$ and after test $\mathit{(Y)}$ where $\mathit{X}$ = {1,1,1,2,2,2,3,3,3} and $\mathit{Y}$ = {1,1,1,2,2,2,3,3,3}. For this example, $\mathit{X = Y}$. The distributions are uniform and the joint probability matrix is given above where the indexes relate to $\mathit{i}$ = 1, $\mathit{j}$ = 2 and $\mathit{k}$ = 3. It follows that $\mathit{H(X) = H(Y) = H(X,Y) = I(X;Y) = 1.585}$. $\endgroup$
    – Mari153
    Commented Sep 11, 2020 at 8:35
  • $\begingroup$ How about a case where $X\neq Y$? $\endgroup$
    – develarist
    Commented Sep 11, 2020 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.