4
$\begingroup$

I am trying to understand maximum entropy modelling and I came across log likelihood equation of the empirical distribution, which I did not quite understand, which also eventually turns out to be equal to the dual function we get when trying to maximise the entropy with constraints using Lagrange's multipliers

$$ L_\widetilde{p}(p) \equiv log \prod\limits_{x,y}p(y|x)^{\widetilde{p}(x,y)} = \sum\limits_{x,y}\widetilde{p}(x,y)\log{p(y|x)}$$

Where,

  • $y$ is the outcome produced by a random process
  • $x$ is the feature influencing the outcome $y$
  • $L_\widetilde{p}(p)$ is the log likelihood
  • $\widetilde{p}$ is the empirical distribution of training data
  • $p(y|x)$ is the model
  • $\widetilde{p}(x,y) \equiv \frac{1}{N} \times \text{number of times that } (x,y) \text{ occurs in the sample} $

Can someone please explain how $p(y|x)$ is raised to the power of $\widetilde{p}(x,y)$ in the log likelihood equation mentioned above? Instead, shouldn't $p(x|y)$ be raised to the power of number of times that $(x,y)$ occurs in the sample.

I went through this reference tutorial on max entropy.

$\endgroup$

1 Answer 1

1
$\begingroup$

As you have already said, if $\tilde p$ is the empirical distribution of the data then by definition $\tilde p(x,y)=\frac{n(x,y)}{N}$ where $N$ is the total sample size and $n(x,y)$ is the number of times $(x,y)$ occurs in the sample. Thus :

$$\log\prod p(x,y)^{\tilde p(x,y)}=\log\left(\prod p(x,y)^{n(x,y)}\right)^{1/N}=\frac{1}{N}\log\prod p(x,y)^{n(x,y)}$$

The quantity on the left is thus proportional to the log likelihood. Since the main goal is maximizing it, the proportionality constant can be skipped. I guess the authors mistakenly call it "log likelihood" which is a minor misnomer.

This quantity is the average log likelihood per observation, that is easier to use because somehow independent of the sample size and easier to compare across datasets of different sizes. Personally I always work with this quantity for this reason and also call it "log likelihood" informally.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.