# MLE and Cross Entropy for Conditional Probabilities

I'm trying to understand the relationship between maximum likelihood estimation for a function of the type $$p(y^{(i)}|x^{(i)};\theta)$$ and the related cross entropy minimization.

For a single variable this is straight forward. I am using the notation from "Deep Learning" by Goodfellow et al.

$$\hat{\theta}_{ML} = \operatorname{argmax}_{\theta} \frac{1}{N}\sum_{i=1}^{N} \log p_{model}(x^{(i)};\theta)$$. Writing the empirical distribution of the data as $$\hat{p}_{data}$$ we can re-write the function inside the argmax as $$E_{\hat{p}_{data}}[\log p_{model}(x;\theta)]$$. Maximizing this function is equivalent to minimizing the cross entropy $$H(\hat{p}_{data},p_{model})$$. Ok, easy enough.

I'm confused how this generalizes to the case where we are maximizing the likelihood of a model of the type $$p_{model}(y^{(i)}|x^{(i)};\theta)$$ as we might do in a supervised learning framework.

Trying to follow the same line of reasoning we should be able to derive a statement that maximizing the likelihood $$\hat{\theta}_{ML} = \operatorname{argmax} \frac{1}{N} \sum_{i=1}^{N} \log p_{model}(y^{(i)}|x^{(i)};\theta)$$ is equivalent to minimizing the cross entropy between $$\hat{p}_{data}(Y|X)$$ and $$p_{model}(Y|X)$$. However $$H(\hat{p}_{data}(Y|X),p_{model}(Y|X))$$ is a random variable w.r.t. $$X$$ so the analogy doesn't hold.

My best guess is that maximizing the likelihood in this scenario is equivalent to minimizing the expected cross entropy between $$\hat{p}_{data}(Y|X)$$ and $$p_{model}(Y|X)$$ taken over the empirical distribution of $$X$$. Written out this would be $$E_{\hat{p}_{data}(X)}[H(\hat{p}_{data}(Y|X),p_{model}(Y|X))]$$.

Any insight would be greatly appreciated.

Actually, I realized that defining the conditional cross-entropy in my previous answer is not even needed, we can directly draw the equivalence between conditional MLE maximization and classical cross entropy minimization: $$\begin{eqnarray} \hat{\theta}_{ML} &=& \arg\max_{\theta} \frac{1}{N} \sum_{i=1}^{N} \log p_{model}(y^{(i)}|x^{(i)};\theta) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}|x^{(i)};\theta) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}|x^{(i)};\theta) - \log p(x^{(i)}) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}|x^{(i)};\theta) - \log p_{model}(x^{(i)}|\theta) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log \left( p_{model}(y^{(i)}|x^{(i)};\theta)p_{model}(x^{(i)}|\theta) \right) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}, x^{(i)}|\theta) \nonumber \\ &\approx& \arg\min_{\theta} \left(\mathbb{E}_{p_{data}(x,y)}\left[ -\log p_{model}(y,x|\theta) \right] \right) \nonumber \\ &=& \arg\min_{\theta} \left(\mathbb{E}_{p_{data}(x,y)}\left[ -\log p_{model(\theta)}(y,x) \right] \right) \nonumber \end{eqnarray}$$ In the third step we subtracted logarithm of true and unknown probability of drawing the sample $$x^{(i)}$$, which does not change the argmax, as it is independent of $$\theta$$. In the fourth step, we defined $$p_{model}(x^{(i)}|\theta) \equiv p(x^{(i)})$$. We can do it, as our model is actually not modeling probability of the sample $$x^{(i)}$$, so artificially defining $$p_{model}(x^{(i)}|\theta) \equiv p(x^{(i)})$$ does not have impact on the model, and is interpreted as "we are modeling the probability of drawing $$x^{(i)}$$ as the true probability of this event (which is actually unknown, but we don't care)". The last step is just a reformulation for those that would irritated by conditioning on $$\theta$$, and the resulting term is just simple cross-entropy minimization.

• Thanks for putting this here. A clear description of this derivation is just what I needed. Jul 20, 2020 at 15:28

I was also wondering about the lack of information on rigorous derivation of the relationship between the conditional MLE (as this can be applied in supervised learning) and cross-entropy minimization on internet. My current explanation follows like this: $$\begin{eqnarray} \hat{\theta}_{ML} &=& \arg\max_{\theta} \frac{1}{N} \sum_{i=1}^{N} \log p_{model}(y^{(i)}|x^{(i)};\theta) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}|x^{(i)};\theta) \nonumber \\ &\approx& \arg\min_{\theta} \left(\mathbb{E}_{p_{data}(x,y)}\left[ -\log p_{model}(y|x;\theta) \right] \right) \nonumber \end{eqnarray}$$ where $$\mathbb{E}_{p_{data}(x,y)}$$ is the expected value with respect to the distribution of the data $$p_{data}(x,y)$$ and the last line can be seen as conditional cross entropy. Note that this term is really rarely used (I found only one paper using it, what wonders me) but this definition of conditional cross-entropy is consistent with the widely used definition of conditional entropy.

The main difference from your approach is, that the expected value is taken over the whole $$X \times Y$$ domain (taking the probability $$p_{data}(x,y)$$ instead of $$p_{data}(y|x)$$), therefore the conditional cross-entropy is not a random variable, but a number.

If you find in this approach any inaccuracies or a better explanation I'll be happy to read about it.

• I also find this is not very well explained in most sources. The cross entropy used as a loss function in DL seems to be implicitly defined like you define it, but I am struggling to find this made explicit in any reference text. I would like to know if you do find a good reference... Jul 18, 2020 at 1:33
• @postylem , the only paper where I found conditional cross-entropy defined is this: arxiv.org/pdf/1809.00197.pdf . Conditional entropy which is similar is used more often and can be also found on wikipedia: en.wikipedia.org/wiki/Conditional_entropy . However, as you can see in my new answer below, I realized defining conditional cross-entropy is actually not even needed, and one can show the equivalence to standard cross entropy minimization directly. Jul 19, 2020 at 10:38

Say the ground truth joint distribution of (data, label) is $$p^*(x,y)$$. This induces a ground truth distribution $$p^*(y|x)$$, which we try to model with $$p_\theta(y|x)$$.
$$\mathbb{E}_{(x, y) \sim p^*(x,y)} [- \log p_\theta(y|x)].$$
This is really just an integral w.r.t. the joint distribution, and by Fubini's theorem we can compute it by two nested integrals (where $$x$$ is held fixed in the inner one), resulting in the conditional cross-entropy, $$\mathbb{E}_{x \sim p^*(x)} [ \mathbb{E}_{y \sim p^*(y|x)} [- \log p_\theta(y|x)] ] = \mathbb{E}_{x \sim p^*(x)} [H[p^*(y|x), p_\theta(y|x)] ].$$