# Entropy based measure of explained variation

I have an observable $$Y$$ which is a function of some set of variables $$\mathcal{X}=\left\{ X_{n}\right\} _{n=1}^{N}$$. Now $$Y$$ is a deterministic function of the $$X_n$$s, but conditioned on only a subset of them, it is not completely determined - and so a "proper" random variable.

I want to express the degree to which $$Y$$ is dependent on any given subset $$A\subseteq\mathcal{X}$$, the amount of "Explained variation" - Something analogous to $$R^2$$ in the case where $$Y$$ is a linear function of the $$X_n$$s.

It seems that the conditional entropy is a natural such measure. i.e. for a subset $$A\subseteq\mathcal{X}$$ the explained variation (lets denote it by $$\rho$$) would be $$\rho\left(A\right) = H\left(Y\right) - H\left(Y\mid A\right)$$ or perhaps normalized to range between 0 and 1 $$\rho\left(A\right) = 1- \frac{H\left(Y\mid A\right)}{H\left(Y\right)}$$

which has a nice interpretation as the amount of surprise left in $$Y$$ after Iv'e observed the variables in $$A$$, as well as the properties one would expect from such a measure - $$\rho\left(\emptyset\right)=0$$, $$\rho\left(\mathcal{X}\right)=1$$ and $$\rho$$ is monotonically increasing with the cardinality of $$A$$

Question: Is this measure used in practice? If so, I'd appreciate a reference. If not, why not?

The first quantity you proposed is actually the mutual information between $$Y$$ and $$A$$:

$$I(Y; A) = H(Y) - H(Y \mid A)$$

This is a central quantity in information theory, and a widely used measure of dependence between random variables. It measures the amount by which knowing $$A$$ reduces uncertainty about $$Y$$ (or the amount by which knowing $$Y$$ reduces uncertainty about $$A$$, since mutual information is symmetric). Equivalently, it can be seen as the KL divergence (a measure of dissimilarity) between the joint distribution and what it would be if $$Y$$ and $$A$$ were independent (the product of the marginals):

$$I(Y; A) = D_{KL} \big( p(Y,A) \parallel p(Y) p(A) \big)$$

A number of normalized variants of mutual information have also been proposed. Your second quantity is actually equal to the uncertainty coefficient $$U(Y \mid A)$$:

$$U(Y \mid A) = \frac{H(Y) - H(Y \mid A)}{H(Y)} = \frac{I(Y; A)}{H(Y)}$$

Also, be aware that estimating mutual information and entropy from data can be challenging, and sometimes even infeasible. This is particularly true with small datasets or in high dimensional settings. A number of estimators have been proposed in the literature; I'd recommend taking a look to get a good grasp of the underlying issues. For example, naive methods like plugging a density estimate in to the entropy formula can be badly biased.

• Hmm, I knew that.. Guess my information-theory is a bit rusty. Well, this is now my favorite intuition for what mutual information is. Thanks! – H.Rappeport Apr 26 at 8:20
• Followup question in the context of using the mutual information as an "explained variation" measure: As I mentioned, $\rho$ is monotonically increasing with the cardinality of $A$, but I'd expect its not like PCA, where you can sort the variables such that for any $n$ the first $n$ variables are the subset of size $n$ with the most explained variation. Are there sufficient conditions on the $X_i$ which allow for such an ordering? I think complete pairwise independence should be enough, but is there some weaker condition? – H.Rappeport Apr 26 at 8:29
• @H.Rappeport. Glad to help. Regarding your followup question, I suspect you're right that such an ordering doesn't always exist. Off the top of my head, not sure about sufficient conditions--you might post this as another question. – user20160 Apr 26 at 19:14