Likelihood of Linear Discriminant Analysis compared to logistic regression

Question

I've come across an interesting exercise. We are given four classification models for binary response and a $d$-dimensional independent variable:

1. A Linear Discriminant Analysis model where the covariance of all classes is the identity matrix.
2. A Quadratic Discriminant Analysis model with unconstrained covariance matrices.
3. A logistic regression model.
4. A logistic regression model with a polynomial basis function expansion of degree 2.

We assume a uniform prior distribution over the two classes.

After training, we inspect the log-likelihood obtained with each model $M$:

$$ \frac{1}{n}\sum_{i=1}^n \mbox{log} p(y_i|\mathbf{x}_i, \boldsymbol{\hat\theta}, M) $$

Given certain pairs of models (e.g. 1 and 3) we are asked to determine if one of them will always perform better with respect to the log-likelihood on the training set and, in that case, which one.

Specifically, the exercise asks about these pairs of models: 1 vs. 3, 2 vs. 4, 3 vs. 4 and 1 vs. 4.

I'm not sure how to approach this. Intuitively I could say, for instance, that given that LDA makes more assumptions than logistic regression, we could say that 3 will always attain a higher optimum than 1 and 2, since it searches within a wider function space.

However, another point of view is to count the number of free parameters of the model. In that case, model 1 would perform better than model 3, with, since the former has $2d$ parameters (the mean of each class) and the latter only has $d+1$, giving the optimizer more room to maximize in the case of 1.

Perhaps I should consider a combination of the two: models 1 and 3 learn a linear function of the input, but model 1 has more free parameters. Models 2 and 4 learn a quadratic function, although the number of free parameters is in general larger in the case of model 2.

What is the right way to approach this problem?

Answer 1

Since it's a self-study question I'll give you a hint to one of the sub questions: LDA has a special property when all classes discriminated between have the exact same covariance matrix. That property should manifest itself on the training data, given the training data actually meets the assumptions, but not necessarily on the test data.