# Likelihood of LDA compared to logistic regression

I've come across an interesting exercise. We are given four classification models for binary response and a $d$-dimensional independent variable:

1. A Linear Discriminant Analysis model where the covariance of all classes is the identity matrix.
2. A Quadratic Discriminant Analysis model with unconstrained covariance matrices.
3. A logistic regression model.
4. A logistic regression model with a polynomial basis function expansion of degree 2.

We assume a uniform prior distribution over the two classes.

After training, we inspect the log-likelihood obtained with each model $M$:

$$\frac{1}{n}\sum_{i=1}^n \mbox{log} p(y_i|\mathbf{x}_i, \boldsymbol{\hat\theta}, M)$$

Given certain pairs of models (e.g. 1 and 3) we are asked to determine if one of them will always perform better with respect to the log-likelihood on the training set and, in that case, which one.

Specifically, the exercise asks about these pairs of models: 1 vs. 3, 2 vs. 4, 3 vs. 4 and 1 vs. 4.

I'm not sure how to approach this. Intuitively I could say, for instance, that given that LDA makes more assumptions than logistic regression, we could say that 3 will always attain a higher optimum than 1 and 2, since it searches within a wider function space.

However, another point of view is to count the number of free parameters of the model. In that case, model 1 would perform better than model 3, with, since the former has $2d$ parameters (the mean of each class) and the latter only has $d+1$, giving the optimizer more room to maximize in the case of 1.

Perhaps I should consider a combination of the two: models 1 and 3 learn a linear function of the input, but model 1 has more free parameters. Models 2 and 4 learn a quadratic function, although the number of free parameters is in general larger in the case of model 2.

What is the right way to approach this problem?

• Likelihood of QDA should be higher than that of LDA, but I am not sure there exists a fixed relationship between likelihoods of LDA and of logistic regression. It might be that it's dataset-dependent. Aug 10 '16 at 0:05
• The relationship between LDA and QDA seems clear, as does the one between logistic regression with different sets of features, given the increased capacity of QDA and LogReg. But yes, the relationship between discriminant analysis and logistic regression is what evades me. Aug 10 '16 at 10:06
• The exercise does consider the possibility that it is data set dependent, mind you. I've edited to add the pairs of models that the exercise asks about specifically. Aug 10 '16 at 10:07
• You can find info about LDA vs logistic regression in book Elements of statistical learning where both are compared in chapter 4. May 5 '20 at 6:01