4
$\begingroup$

Suppose we have two random variables $X$ and $Y$ (for simplicity of exposition I will take these to be discrete). If we were to condition our entire analysis on the event $X=x$ and then ask for the entropy of $Y$ under this restriction on the sample space then the resulting entropy (which I'll call the "quasi-conditional entropy" for now) would be the conditional expectation:

$$h(Y|X=x) \equiv \mathbb{E}( -\log p(Y|X) | X=x) = -\sum_{y \in \mathscr{Y}} p(y|x) \log(p(y|x)). \ \quad \quad \quad \quad$$

However, in information theory, the conditional entropy of $Y$ given $X$ is actually defined as the marginal expectation:

$$\begin{align} \quad \quad \quad H(Y|X) \equiv \mathbb{E}(-\log p(Y|X)) &= - \sum_{x \in \mathscr{X}} \sum_{y \in \mathscr{Y}} p(x,y) \log p(y|x) \\[6pt] &= - \sum_{x \in \mathscr{X}} p(x) \sum_{y \in \mathscr{Y}} p(y|x) \log p(y|x) \\[6pt] &= - \sum_{x \in \mathscr{X}} p(x) \cdot h(Y|X=x). \\[6pt] \end{align}$$

Unlike other "conditional" quantities used in statistics, this quantity is not a function of the conditioning variable, since this part is "marginalised out" in the definition. Now, there are a bunch of good reasons why this quantity is useful in information theory, so let's accept that this latter definition is the correct meaning of the "conditional entropy" in this situation, notwithstanding that it is not a function of the conditioning variable. This terminology is a bit annoying, but I can live with it. Still, it raises the obvious question: if not the "conditional entropy", what do we call the function $h(Y|X=x)$?

I've seen various resources on information theory and entropy that use the function $h(Y|X=x)$ (often for the intermediate step in computing the conditional entropy above) but I've not seen it named. In a sense it is a conditional entropy ---since it is the entropy of a random variable when we condition the entire analysis on some other random variable--- but we can't use this name because it is already taken by $H(Y|X)$.


Question: What is the proper name for the function $h(Y|X=x)$ in this context? Is there a standard name for this function in the information theory/statistical literature? Are there any alternative characterisations of information theory that call the function $h(Y|X=x)$ the "conditional entropy" and call $H(Y|X)$ something else?

$\endgroup$
2
  • $\begingroup$ Since $h(Y|X=x)$ is only condition on the single event, and $H(Y|X)$ is weighted sum of all possible $h$ values (at least according to Wikipedia entry), then $h$ is not qualified to be called conditional entropy. "Quasi-conditional entropy" is also not a good name while quasi usually implies a static approximation to a dynamic process. Maybe "marginal (Shannon) entropy" could differentiate this. $\endgroup$ May 29, 2022 at 18:27
  • $\begingroup$ Yes, indeed; the question makes the same point. But in practice it is sometimes called "conditional" so I am seeking a better name. Thanks for the name suggestion! $\endgroup$
    – Ben
    May 29, 2022 at 23:52

2 Answers 2

4
$\begingroup$

I'm not aware of a standard term, and agree that it's slightly annoying. A quick glance at Cover & Thomas seems to confirm that they don't discuss this. Wikipedia calls $H(Y \mid X=x)$ the "entropy of $Y$ conditioned on $X$ taking the value $x$". I personally tend to use something like "the entropy of the conditional distribution" (when it's clear which distribution we're talking about) or "the conditional entropy of $Y$ given $X=x$". The latter is similar (modulo capitalization) to MacKay's terminology in Information theory, inference, and learning algorithms. From section 8.1:

The conditional entropy of $X$ given $y=b_k$ is the entropy of the probability distribution $P(X \mid y=b_k).$

$$H(X \mid y=b_k) \equiv \sum_{x \in \mathcal{A}_X} P(x \mid y=b_k) \log \frac{1}{P(x \mid y=b_k)} \tag{8.3}$$

In the next sentence, he also uses the apparent shorthand "conditional entropy of $X$ given $y$":

The conditional entropy of $X$ given $Y$ is the average, over $y$, of the conditional entropy of $X$ given $y$.

$$H(X \mid Y) \equiv \sum_{y \in \mathcal{A}_Y} P(y) \left[ \sum_{x \in \mathcal{A}_X} P(x \mid y) \log \frac{1}{P(x \mid y)} \right] \tag{8.4}$$

But, I think this shorthand risks confusion because it relies on a small difference in capitalization ($Y$ vs. $y$) to indicate a big difference in meaning (whether or not we're averaging over $y$ values).

$\endgroup$
-1
$\begingroup$

The quantity:

$$\begin{align} \quad \quad \quad H(Y|X) \equiv \mathbb{E}(-\log p(Y|X)) &= - \sum_{x \in \mathscr{X}} \sum_{y \in \mathscr{Y}} p(x,y) \log p(y|x) \\[6pt] &= - \sum_{x \in \mathscr{X}} p(x) \sum_{y \in \mathscr{Y}} p(y|x) \log p(y|x) \\[6pt] &= - \sum_{x \in \mathscr{X}} p(x) \cdot h(Y|X=x). \\[6pt] \end{align}$$

is the conditional entropy of the experiment $Y$ calculated on the assumption that the event $x$ of the experiment $X$ occurred. In other word, it is the entropy of the experiment $Y$ conditioned by the outcome $x$. The quantity:

$$\begin{align} \quad \quad \quad H(Y|X) \equiv \mathbb{E}(-\log p(Y|X)) &= - \sum_{x \in \mathscr{X}} \sum_{y \in \mathscr{Y}} p(x,y) \log p(y|x) \\[6pt] &= - \sum_{x \in \mathscr{X}} p(x) \sum_{y \in \mathscr{Y}} p(y|x) \log p(y|x) \\[6pt] &= - \sum_{x \in \mathscr{X}} p(x) \cdot h(Y|X=x). \\[6pt] \end{align}$$

is the entropy of the experiment $Y$ conditioned to the experiment $X$. Stricly speaking, the term conditional entropy is not correct in this case, although, as you pointed out, in the literature, the term conditional entropy is often referred to as the marginal expectation. The term conditional entropy is misused because there is no exact name for the entropy of the experiment $Y$ conditioned on the experiment $X$, necessitating a circumlocution. You can find the exact definition in the text "Information Theory with Applications" by Silviu Guiasu.

New contributor
Upax is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.