I have an observable $Y$ which is a function of some set of variables $\mathcal{X}=\left\{ X_{n}\right\} _{n=1}^{N}$. Now $Y$ is a deterministic function of the $X_n$s, but conditioned on only a subset of them, it is not completely determined - and so a "proper" random variable.

I want to express the degree to which $Y$ is dependent on any given subset $A\subseteq\mathcal{X}$, the amount of "Explained variation" - Something analogous to $R^2$ in the case where $Y$ is a linear function of the $X_n$s.

It seems that the conditional entropy is a natural such measure. i.e. for a subset $A\subseteq\mathcal{X}$ the explained variation (lets denote it by $\rho$) would be $$\rho\left(A\right) = H\left(Y\right) - H\left(Y\mid A\right)$$ or perhaps normalized to range between 0 and 1 $$\rho\left(A\right) = 1- \frac{H\left(Y\mid A\right)}{H\left(Y\right)}$$

which has a nice interpretation as the amount of surprise left in $Y$ after Iv'e observed the variables in $A$, as well as the properties one would expect from such a measure - $\rho\left(\emptyset\right)=0$, $\rho\left(\mathcal{X}\right)=1$ and $\rho$ is monotonically increasing with the cardinality of $A$

Question: Is this measure used in practice? If so, I'd appreciate a reference. If not, why not?


The first quantity you proposed is actually the mutual information between $Y$ and $A$:

$$I(Y; A) = H(Y) - H(Y \mid A)$$

This is a central quantity in information theory, and a widely used measure of dependence between random variables. It measures the amount by which knowing $A$ reduces uncertainty about $Y$ (or the amount by which knowing $Y$ reduces uncertainty about $A$, since mutual information is symmetric). Equivalently, it can be seen as the KL divergence (a measure of dissimilarity) between the joint distribution and what it would be if $Y$ and $A$ were independent (the product of the marginals):

$$I(Y; A) = D_{KL} \big( p(Y,A) \parallel p(Y) p(A) \big)$$

A number of normalized variants of mutual information have also been proposed. Your second quantity is actually equal to the uncertainty coefficient $U(Y \mid A)$:

$$U(Y \mid A) = \frac{H(Y) - H(Y \mid A)}{H(Y)} = \frac{I(Y; A)}{H(Y)}$$

Also, be aware that estimating mutual information and entropy from data can be challenging, and sometimes even infeasible. This is particularly true with small datasets or in high dimensional settings. A number of estimators have been proposed in the literature; I'd recommend taking a look to get a good grasp of the underlying issues. For example, naive methods like plugging a density estimate in to the entropy formula can be badly biased.

| cite | improve this answer | |
  • $\begingroup$ Hmm, I knew that.. Guess my information-theory is a bit rusty. Well, this is now my favorite intuition for what mutual information is. Thanks! $\endgroup$ – H.Rappeport Apr 26 at 8:20
  • $\begingroup$ Followup question in the context of using the mutual information as an "explained variation" measure: As I mentioned, $\rho$ is monotonically increasing with the cardinality of $A$, but I'd expect its not like PCA, where you can sort the variables such that for any $n$ the first $n$ variables are the subset of size $n$ with the most explained variation. Are there sufficient conditions on the $X_i$ which allow for such an ordering? I think complete pairwise independence should be enough, but is there some weaker condition? $\endgroup$ – H.Rappeport Apr 26 at 8:29
  • $\begingroup$ @H.Rappeport. Glad to help. Regarding your followup question, I suspect you're right that such an ordering doesn't always exist. Off the top of my head, not sure about sufficient conditions--you might post this as another question. $\endgroup$ – user20160 Apr 26 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.