What is the proper name for the *other* type of "conditional entropy"?

Question

Suppose we have two random variables $X$ and $Y$ (for simplicity of exposition I will take these to be discrete). If we were to condition our entire analysis on the event $X=x$ and then ask for the entropy of $Y$ under this restriction on the sample space then the resulting entropy (which I'll call the "quasi-conditional entropy" for now) would be the conditional expectation:

$$h(Y|X=x) \equiv \mathbb{E}( -\log p(Y|X) | X=x) = -\sum_{y \in \mathscr{Y}} p(y|x) \log(p(y|x)). \ \quad \quad \quad \quad$$

However, in information theory, the conditional entropy of $Y$ given $X$ is actually defined as the marginal expectation:

$$\begin{align} \quad \quad \quad H(Y|X) \equiv \mathbb{E}(-\log p(Y|X)) &= - \sum_{x \in \mathscr{X}} \sum_{y \in \mathscr{Y}} p(x,y) \log p(y|x) \\[6pt] &= - \sum_{x \in \mathscr{X}} p(x) \sum_{y \in \mathscr{Y}} p(y|x) \log p(y|x) \\[6pt] &= - \sum_{x \in \mathscr{X}} p(x) \cdot h(Y|X=x). \\[6pt] \end{align}$$

Unlike other "conditional" quantities used in statistics, this quantity is not a function of the conditioning variable, since this part is "marginalised out" in the definition. Now, there are a bunch of good reasons why this quantity is useful in information theory, so let's accept that this latter definition is the correct meaning of the "conditional entropy" in this situation, notwithstanding that it is not a function of the conditioning variable. This terminology is a bit annoying, but I can live with it. Still, it raises the obvious question: if not the "conditional entropy", what do we call the function $h(Y|X=x)$?

I've seen various resources on information theory and entropy that use the function $h(Y|X=x)$ (often for the intermediate step in computing the conditional entropy above) but I've not seen it named. In a sense it is a conditional entropy ---since it is the entropy of a random variable when we condition the entire analysis on some other random variable--- but we can't use this name because it is already taken by $H(Y|X)$.

---

Question: What is the proper name for the function $h(Y|X=x)$ in this context? Is there a standard name for this function in the information theory/statistical literature? Are there any alternative characterisations of information theory that call the function $h(Y|X=x)$ the "conditional entropy" and call $H(Y|X)$ something else?

Answer 1

I'm not aware of a standard term, and agree that it's slightly annoying. A quick glance at Cover & Thomas seems to confirm that they don't discuss this. Wikipedia calls $H(Y \mid X=x)$ the "entropy of $Y$ conditioned on $X$ taking the value $x$". I personally tend to use something like "the entropy of the conditional distribution" (when it's clear which distribution we're talking about) or "the conditional entropy of $Y$ given $X=x$". The latter is similar (modulo capitalization) to MacKay's terminology in Information theory, inference, and learning algorithms. From section 8.1:

The conditional entropy of $X$ given $y=b_k$ is the entropy of the probability distribution $P(X \mid y=b_k).$

$$H(X \mid y=b_k) \equiv \sum_{x \in \mathcal{A}_X} P(x \mid y=b_k) \log \frac{1}{P(x \mid y=b_k)} \tag{8.3}$$

In the next sentence, he also uses the apparent shorthand "conditional entropy of $X$ given $y$":

The conditional entropy of $X$ given $Y$ is the average, over $y$, of the conditional entropy of $X$ given $y$.

$$H(X \mid Y) \equiv \sum_{y \in \mathcal{A}_Y} P(y) \left[ \sum_{x \in \mathcal{A}_X} P(x \mid y) \log \frac{1}{P(x \mid y)} \right] \tag{8.4}$$

But, I think this shorthand risks confusion because it relies on a small difference in capitalization ($Y$ vs. $y$) to indicate a big difference in meaning (whether or not we're averaging over $y$ values).