# Mutual information relationship to copula entropy is borked?

In the paper, they say that mutual information $$I(x)$$ is related to copula entropy $$H_c(x)$$ via $$I(x) = -H_c(x)$$.

However, the sign of entropy is supposed to be non-negative, as is the sign of mutual information. This leads me to conclude that mutual information and copula entropy are zero. If mutual information is zero, then all variables are unrelated, so something here is borked.

In the linked question, ArnoV commented that differential entropy can be either positive or negative (or zero), and the R package by the paper authors appears to be for continuous variables (it does not like having tied values). Fine. I even have heard that copula theory gets a bit weird for discrete marginal variables (copulae are not unique), so perhaps we should restrict ourselves to the continuous case. Okay...

Then $$H_c(x)\in\mathbb{R}$$; entropy can be any real value. This then means that mutual information can be any real value.

But mutual information is not negative! This leads right back to both mutual information and copula entropy having to be zero. Contradiction! $$\Rightarrow\Leftarrow$$

The result in the paper is elegant and makes a lot of sense to me. If the copula describes the relationship between marginal variables and the mutual information quantifies the degree to which the marginal variables are dependent, they should be related. Yet the equation they give seems to result in math being broken.

What's going on? What am I missing?

• I have also read that continuity of the distribution function(s) will guarantee that the copula is unique. Commented Oct 3, 2021 at 3:31
• I don't know if this is related, but differential entropy is not the continuous analogue of Shannon's entropy. Shannon assumed this to be the case in his original work, but it is not the case. Commented Oct 3, 2021 at 3:34
• MI is not entropy: it can be expressed as differences of entropies.. There is no paradox here. It isn't any different than wondering why it's possible to locate points with negative values on a number line, despite the fact that all distances between such points are non-negative.
– whuber
Commented Jan 1, 2022 at 22:02

As you already mentioned, differential entropy may become negative and that is exactly what happens here. Recap that differential entropy is some measure for the uncertainty or randomness of a distribution. Given that we have a multivariate model for $$X_1$$ and $$X_2$$, we encounter the greatest uncertainty if $$X_1$$ and $$X_2$$ are independent, as on a quantile level, the knowledge of $$X_1$$ does not yield any information about the quantiles of $$X_2$$ and vice versa. The copula in this case would be the independence coupla with $$C(u,v)=u\cdot v$$ and density $$c(u,v)=1$$. The copula entropy would be then:
$$$$H_c=-\int_{[0,1]^2}c(u,v)\log(c(u,v))d(u,v)=-\int_{[0,1]^2}1\cdot\log(1)d(u,v)=0,$$$$
as $$\log(1)=0$$ and the mutual information would be zero as well. If however on the other hand, $$X_1$$ and $$X_2$$ are dependent, we are able to infer from one variable about the other and that's exactly the point where some of the randomness of the model get's lost, and thus the total entropy declines. As we however, did not make any (changing) assumptions considering the model's marginal distributions, it's also on an intuitive level plausible that copula entropy is negative (or zero).