MLE and Cross Entropy for Conditional Probabilities

Question

I'm trying to understand the relationship between maximum likelihood estimation for a function of the type $p(y^{(i)}|x^{(i)};\theta)$ and the related cross entropy minimization.

For a single variable this is straight forward. I am using the notation from "Deep Learning" by Goodfellow et al.

$\hat{\theta}_{ML} = \operatorname{argmax}_{\theta} \frac{1}{N}\sum_{i=1}^{N} \log p_{model}(x^{(i)};\theta)$. Writing the empirical distribution of the data as $\hat{p}_{data}$ we can re-write the function inside the argmax as $E_{\hat{p}_{data}}[\log p_{model}(x;\theta)]$. Maximizing this function is equivalent to minimizing the cross entropy $H(\hat{p}_{data},p_{model})$. Ok, easy enough.

I'm confused how this generalizes to the case where we are maximizing the likelihood of a model of the type $p_{model}(y^{(i)}|x^{(i)};\theta)$ as we might do in a supervised learning framework.

Trying to follow the same line of reasoning we should be able to derive a statement that maximizing the likelihood $\hat{\theta}_{ML} = \operatorname{argmax} \frac{1}{N} \sum_{i=1}^{N} \log p_{model}(y^{(i)}|x^{(i)};\theta)$ is equivalent to minimizing the cross entropy between $\hat{p}_{data}(Y|X)$ and $p_{model}(Y|X)$. However $H(\hat{p}_{data}(Y|X),p_{model}(Y|X))$ is a random variable w.r.t. $X$ so the analogy doesn't hold.

My best guess is that maximizing the likelihood in this scenario is equivalent to minimizing the expected cross entropy between $\hat{p}_{data}(Y|X)$ and $p_{model}(Y|X)$ taken over the empirical distribution of $X$. Written out this would be $E_{\hat{p}_{data}(X)}[H(\hat{p}_{data}(Y|X),p_{model}(Y|X))]$.

Any insight would be greatly appreciated.

Answer 1

Actually, I realized that defining the conditional cross-entropy in my previous answer is not even needed, we can directly draw the equivalence between conditional MLE maximization and classical cross entropy minimization: \begin{eqnarray} \hat{\theta}_{ML} &=& \arg\max_{\theta} \frac{1}{N} \sum_{i=1}^{N} \log p_{model}(y^{(i)}|x^{(i)};\theta) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}|x^{(i)};\theta) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}|x^{(i)};\theta) - \log p(x^{(i)}) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}|x^{(i)};\theta) - \log p_{model}(x^{(i)}|\theta) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log \left( p_{model}(y^{(i)}|x^{(i)};\theta)p_{model}(x^{(i)}|\theta) \right) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}, x^{(i)}|\theta) \nonumber \\ &\approx& \arg\min_{\theta} \left(\mathbb{E}_{p_{data}(x,y)}\left[ -\log p_{model}(y,x|\theta) \right] \right) \nonumber \\ &=& \arg\min_{\theta} \left(\mathbb{E}_{p_{data}(x,y)}\left[ -\log p_{model(\theta)}(y,x) \right] \right) \nonumber \end{eqnarray} In the third step we subtracted logarithm of true and unknown probability of drawing the sample $x^{(i)}$, which does not change the argmax, as it is independent of $\theta$. In the fourth step, we defined $p_{model}(x^{(i)}|\theta) \equiv p(x^{(i)})$. We can do it, as our model is actually not modeling probability of the sample $x^{(i)}$, so artificially defining $p_{model}(x^{(i)}|\theta) \equiv p(x^{(i)})$ does not have impact on the model, and is interpreted as "we are modeling the probability of drawing $x^{(i)}$ as the true probability of this event (which is actually unknown, but we don't care)". The last step is just a reformulation for those that would irritated by conditioning on $\theta$, and the resulting term is just simple cross-entropy minimization.

Answer 2

I was also wondering about the lack of information on rigorous derivation of the relationship between the conditional MLE (as this can be applied in supervised learning) and cross-entropy minimization on internet. My current explanation follows like this: $$\begin{eqnarray} \hat{\theta}_{ML} &=& \arg\max_{\theta} \frac{1}{N} \sum_{i=1}^{N} \log p_{model}(y^{(i)}|x^{(i)};\theta) \nonumber \\ &=& \arg\min_{\theta} \frac{1}{N} \sum_{i=1}^{N} - \log p_{model}(y^{(i)}|x^{(i)};\theta) \nonumber \\ &\approx& \arg\min_{\theta} \left(\mathbb{E}_{p_{data}(x,y)}\left[ -\log p_{model}(y|x;\theta) \right] \right) \nonumber \end{eqnarray}$$ where $\mathbb{E}_{p_{data}(x,y)}$ is the expected value with respect to the distribution of the data $p_{data}(x,y)$ and the last line can be seen as conditional cross entropy. Note that this term is really rarely used (I found only one paper using it, what wonders me) but this definition of conditional cross-entropy is consistent with the widely used definition of conditional entropy.

The main difference from your approach is, that the expected value is taken over the whole $X \times Y$ domain (taking the probability $p_{data}(x,y)$ instead of $p_{data}(y|x)$), therefore the conditional cross-entropy is not a random variable, but a number.

If you find in this approach any inaccuracies or a better explanation I'll be happy to read about it.

Answer 3

Your guess is correct.

Say the ground truth joint distribution of (data, label) is $p^*(x,y)$. This induces a ground truth distribution $p^*(y|x)$, which we try to model with $p_\theta(y|x)$.

Let's just suppose an infinite amount of samples are available, so all our Monte-Carlo approximations are exact. Then the conditional MLE objective is

$$ \mathbb{E}_{(x, y) \sim p^*(x,y)} [- \log p_\theta(y|x)]. $$

This is really just an integral w.r.t. the joint distribution, and by Fubini's theorem we can compute it by two nested integrals (where $x$ is held fixed in the inner one), resulting in the conditional cross-entropy, $$ \mathbb{E}_{x \sim p^*(x)} [ \mathbb{E}_{y \sim p^*(y|x)} [- \log p_\theta(y|x)] ] = \mathbb{E}_{x \sim p^*(x)} [H[p^*(y|x), p_\theta(y|x)] ]. $$

As @Igor remarked, this conditional cross-entropy takes a similar form as the conditional entropy.

Also see Appendix of B of this note by Karl Stratos.