Mutual information relationship to copula entropy is borked?

Question

I have posted a related Question based on a paper, Ma, Jian, and Zengqi Sun. "Mutual information is copula entropy." Tsinghua Science & Technology 16.1 (2011): 51-54.

In the paper, they say that mutual information $I(x)$ is related to copula entropy $H_c(x)$ via $I(x) = -H_c(x)$.

However, the sign of entropy is supposed to be non-negative, as is the sign of mutual information. This leads me to conclude that mutual information and copula entropy are zero. If mutual information is zero, then all variables are unrelated, so something here is borked.

In the linked question, ArnoV commented that differential entropy can be either positive or negative (or zero), and the R package by the paper authors appears to be for continuous variables (it does not like having tied values). Fine. I even have heard that copula theory gets a bit weird for discrete marginal variables (copulae are not unique), so perhaps we should restrict ourselves to the continuous case. Okay...

Then $H_c(x)\in\mathbb{R}$; entropy can be any real value. This then means that mutual information can be any real value.

But mutual information is not negative! This leads right back to both mutual information and copula entropy having to be zero. Contradiction! $\Rightarrow\Leftarrow$

The result in the paper is elegant and makes a lot of sense to me. If the copula describes the relationship between marginal variables and the mutual information quantifies the degree to which the marginal variables are dependent, they should be related. Yet the equation they give seems to result in math being broken.

What's going on? What am I missing?

Answer 1

I'll try to provide some intuition:

As you already mentioned, differential entropy may become negative and that is exactly what happens here. Recap that differential entropy is some measure for the uncertainty or randomness of a distribution. Given that we have a multivariate model for $X_1$ and $X_2$, we encounter the greatest uncertainty if $X_1$ and $X_2$ are independent, as on a quantile level, the knowledge of $X_1$ does not yield any information about the quantiles of $X_2$ and vice versa. The copula in this case would be the independence coupla with $C(u,v)=u\cdot v$ and density $c(u,v)=1$. The copula entropy would be then:

\begin{equation} H_c=-\int_{[0,1]^2}c(u,v)\log(c(u,v))d(u,v)=-\int_{[0,1]^2}1\cdot\log(1)d(u,v)=0, \end{equation}

as $\log(1)=0$ and the mutual information would be zero as well. If however on the other hand, $X_1$ and $X_2$ are dependent, we are able to infer from one variable about the other and that's exactly the point where some of the randomness of the model get's lost, and thus the total entropy declines. As we however, did not make any (changing) assumptions considering the model's marginal distributions, it's also on an intuitive level plausible that copula entropy is negative (or zero).