# Likelihood of Linear Discriminant Analysis compared to logistic regression

I've come across an interesting exercise. We are given four classification models for binary response and a $d$-dimensional independent variable:

1. A Linear Discriminant Analysis model where the covariance of all classes is the identity matrix.
2. A Quadratic Discriminant Analysis model with unconstrained covariance matrices.
3. A logistic regression model.
4. A logistic regression model with a polynomial basis function expansion of degree 2.

We assume a uniform prior distribution over the two classes.

After training, we inspect the log-likelihood obtained with each model $M$:

$$\frac{1}{n}\sum_{i=1}^n \mbox{log} p(y_i|\mathbf{x}_i, \boldsymbol{\hat\theta}, M)$$

Given certain pairs of models (e.g. 1 and 3) we are asked to determine if one of them will always perform better with respect to the log-likelihood on the training set and, in that case, which one.

Specifically, the exercise asks about these pairs of models: 1 vs. 3, 2 vs. 4, 3 vs. 4 and 1 vs. 4.

I'm not sure how to approach this. Intuitively I could say, for instance, that given that LDA makes more assumptions than logistic regression, we could say that 3 will always attain a higher optimum than 1 and 2, since it searches within a wider function space.

However, another point of view is to count the number of free parameters of the model. In that case, model 1 would perform better than model 3, with, since the former has $2d$ parameters (the mean of each class) and the latter only has $d+1$, giving the optimizer more room to maximize in the case of 1.

Perhaps I should consider a combination of the two: models 1 and 3 learn a linear function of the input, but model 1 has more free parameters. Models 2 and 4 learn a quadratic function, although the number of free parameters is in general larger in the case of model 2.

What is the right way to approach this problem?

• Likelihood of QDA should be higher than that of LDA, but I am not sure there exists a fixed relationship between likelihoods of LDA and of logistic regression. It might be that it's dataset-dependent. Commented Aug 10, 2016 at 0:05
• The relationship between LDA and QDA seems clear, as does the one between logistic regression with different sets of features, given the increased capacity of QDA and LogReg. But yes, the relationship between discriminant analysis and logistic regression is what evades me. Commented Aug 10, 2016 at 10:06
• I can see no reason for any relationship, since the two lokelihoods (for LR and LDA) are defined with respect to different dominating measures ... Commented Oct 26, 2022 at 19:13
• If you use Bayes' rule on LDA you get exactly binary logistic regression. Commented Mar 16 at 12:31
• @kjetilbhalvorsen they are looking at $Y|X$ as the 'likelihood', so it's always with respect to counting measure on $\{0,1\}^n$. Commented Jul 17 at 21:13