Multivariate conditional entropy I would like to take data columns and compute the multivariate conditional entropy.  For instance, suppose I have columns $A, B, C D, E$ and I want to compute the conditional entropy $H(E | A,B,C,D)$.  Can someone give a little explanation of all this, pointing out the first joint variable $A,B,C,D$ vs the second variable $E$.  Then please give the algorithm and math for calculating the conditional entropy.
 A: Definition of conditional entropy
Suppose $A,B,C,D,E$ are random variables with joint distribution $p(a,b,c,d,e)$. The conditional entropy of $E$ given $A,B,C,D$ is defined as the expected value (over the joint distribution) of $-\log p(e \mid a,b,c,d)$:
$$H(E \mid A, B, C, D) =
-\mathbb{E}_{p(a,b,c,d,e)} \big[ \log p(e \mid a, b, c, d) \big]
\tag{1}$$
For discrete random variables, this is:
$$H(E \mid A,B,C,D) =
-\sum_{a,b,c,d,e} p(a,b,c,d,e) \log p(e \mid a,b,c,d)
\tag{2}$$
where the sum is over all possible values taken by the variables. For continuous random variables, simply replace the sum with an integral.
The conditional entropy can also be expressed as the difference between the joint entropy of all variables and the joint entropy of the variables upon which we wish to condition:
$$H(E \mid A,B,C,D) = H(A,B,C,D,E) - H(A,B,C,D) \tag{3}$$
Estimation of conditional entropy
If the joint distribution is known, the conditional entropy can be computed following the definition above. But, if we only have access to data sampled from the joint distribution, the conditional entropy must be estimated.
A straightforward option is to estimate the required probability distributions from the data (e.g. using histograms or kernel density estimates), then plug them into the expressions above. This is called a plug-in estimate. For example, we could estimate $p(a,b,c,d,e)$ and $p(a,b,c,d)$, plug them into the joint entropy formula, then take the difference as in expression $(3)$.
However, plug-in entropy estimators suffer from bias, and are inaccurate when there's not much data available. The problem quickly becomes worse as the number of variables increases. Many improved entropy estimators have been proposed. See Beirlant et al. (2001) for a review, as well as more recent work (example references below). These improved procedures can be used to estimate $H(A,B,C,D,E)$ and $H(A,B,C,D)$ from the data. Then, $H(E \mid A,B,C,D)$ is given by the difference, as in expression $(3)$.
References


*

*Beirlant, Dudewicz, Gyorfi, van der Meulen (1997). Nonparametric entropy estimation: An overview.

*Nemenman, Shafee, Bialek (2002). Entropy and Inference, Revisited.

*Paninski (2003). Estimation of entropy and mutual information.

*Miller (2003). A new class of entropy estimators for multi-dimensional densities.

*Schurmann (2004). Bias analysis in entropy estimation.

*Bonachela, Hinrichsen, Munoz (2008). Entropy estimates of small data sets.

