# MLE vs MAP vs conditional MLE with regards to logistic regression

We have some set of iid RV's: $(X_i, Y_i), \; i=1,\ldots n$.

We believe each to be distributed as $P(X_i, Y_i | \theta)$. So that $$P(X,Y | \theta) = \prod_i P_i(X_i, Y_i | \theta)$$ Now using Baye's rule: $$P(\theta|X,Y) = \frac{P(X,Y|\theta)P(\theta)}{P(X,Y)} = \frac{P(\theta)\prod_i P_i(X_i, Y_i | \theta)}{P(X,Y)}$$

As I understand it, MLE, MAP, and conditional MLE all attempt to find the best parameters, $\theta$, given the data by maximizing the left hand side by maximizing a subset of terms on the right.

For MLE, we maximize the likelihood term, $\prod_i P_i(X_i, Y_i | \theta)$.

For MAP, we maximize all of the numerator, $P(\theta)\prod_i P_i(X_i, Y_i | \theta)$.

For conditional MLE (as in logistic regression), we have $$\frac{P(\theta)\prod_i P_i(X_i, Y_i | \theta)}{P(X,Y)} = \frac{P(\theta) \left( \prod_i P_i(Y_i | X_i, \theta) \right) \left( \prod_i P(X_i|\theta) \right) }{P(X,Y)}$$

Conditional MLE maximizes only the $\prod_i P_i(Y_i | X_i, \theta)$ term.

Is this correct? I have seen regularized logistic regression amounting to maximizing the prior, $P(\theta)$. Would modeling the third distribution for conditional MLE, $\prod_i P(X_i|\theta)$, be something different entirely?

I understand that logistic regression is discriminative model. Is this a result of this? Would modeling $P(X_i|\theta)$ then give us a generative model?

Thanks for any pointers.

## 1 Answer

For MLE and MAP you are right. "Conditional MLE" is another way of saying "MLE in a conditional model". Logistic regression is a conditional model in the sense that $\theta$ only controls $P(Y|X)$ (it has no effect on $P(X)$). Therefore the MLE for logistic regression is a conditional MLE. If you have a model in which $\theta$ affects $P(X)$ then it is no longer a conditional model. The MLE in such a model cannot be regarded as a conditional MLE. A discriminative model is the same thing as a conditional model.

The Wikipedia pages for generative model and discriminative model do a reasonably good job of explaining this, however the definitions there do not correctly handle the case when $P(X)$ exists but does not depend on $\theta$. Wikipedia would say that such a model is generative, even though it would behave in every way like a discriminative model. I would regard such a model as discriminative. For more explanation, see Discriminative models, not discriminative training.

• I think I see. So essentially, we can split the parameters $\theta, \theta'$ into those that those for which $X|\theta'$ depends and for which $X|Y,\theta$ depends conditionally. Furthermore we can assume independence of these parameters $\theta, \theta'$. Finally, one particular case of this is $\theta'$ empty, so that $X$ has no dependence on parameters. Am I interpreting your report correctly? – Chester Sep 13 '14 at 13:21
• Yes, you interpret it correctly. – Tom Minka Sep 13 '14 at 19:20
• Can you give a reason why $X$ does not depend on $\theta$ – user1825567 Sep 1 '18 at 14:01