What's the difference between Maximizing Conditional (Log) Likelihood or Joint (Log) Likelihood while estimating parameters of a model?

Question

Consider a response y and data matrix X. Suppose I'm creating a model of the form -

y ~ g(X,$\theta$)

(g() could be any function of X and $\theta$)

Now, for estimating $\theta$ using Maximum Likelihood (ML) method, I could go ahead either with Conditional ML (assuming I know the form of conditional density f(y|X) ) or with Joint ML (assuming I know the form of joint density f(y,X) or equivalently, f(X|y) * f(y) )

I was wondering if there are any considerations in going ahead with either of the above two methods apart from the assumption about the densities. Also, are there any instances (specific types of data) where one method overpowers other most of the time?

Answer 1

It depends what you want to do with your model later.

Joint models attempt to predict the whole distribution over $X$ and $y$. It has some useful properties:

• Outlier detection. Samples very unlike your training samples can be identified as they'll have a low marginal probability. A conditional model won't necessarily be bale to tell you this.
• Sometimes it's easier to optimise. If your model was a gaussian mixture model, say, there are well documented ways to fit it to the joint density you can just plug in (expectation maximisation, variational bayes), but things get more complicated it you want to train it conditionally.
• Depending on the model, training can potentially be parallelised by taking advantages of conditional independences, and you may also avoid the need to retrain it later if new data becomes available. E.G. if every marginal distribution $f(X|y)$ is parameterised separately, and you observe a new sample $(X=x_1,y=y_1)$, then the only marginal distribution you need to retrain is $f(X|y=y_1)$. The other marginal distributions $f(X|y=y_2), f(X|y=y_3), \ldots$ are unaffected. This property is less common with conditional models.
• I recall reading a paper which indicated joint models have some other nice properties in cases where there's lots and lots of data, but cannot remember the exact claim, or find it in my big folder of interesting papers. If I find it later I'll put in a reference.

Conditional models however have some interesting properties too

• They can work really well.
• Some have had a lot of work put in finding sensible optimisation strategies (e.g. support vector machines)
• The conditional distribution is very often `simpler' to model than the joint - to model the latter, you have to model the former as well as modelling the marginal distribution. If you're only interested in getting accurate predictions of what value $y$ is for a given $X$, it can be more sensible to concentrate your model's capacity on representing this alone.