4
$\begingroup$

I've come across an interesting exercise. We are given four classification models for binary response and a $d$-dimensional independent variable:

  1. A Linear Discriminant Analysis model where the covariance of all classes is the identity matrix.
  2. A Quadratic Discriminant Analysis model with unconstrained covariance matrices.
  3. A logistic regression model.
  4. A logistic regression model with a polynomial basis function expansion of degree 2.

We assume a uniform prior distribution over the two classes.

After training, we inspect the log-likelihood obtained with each model $M$:

$$ \frac{1}{n}\sum_{i=1}^n \mbox{log} p(y_i|\mathbf{x}_i, \boldsymbol{\hat\theta}, M) $$

Given certain pairs of models (e.g. 1 and 3) we are asked to determine if one of them will always perform better with respect to the log-likelihood on the training set and, in that case, which one.

Specifically, the exercise asks about these pairs of models: 1 vs. 3, 2 vs. 4, 3 vs. 4 and 1 vs. 4.

I'm not sure how to approach this. Intuitively I could say, for instance, that given that LDA makes more assumptions than logistic regression, we could say that 3 will always attain a higher optimum than 1 and 2, since it searches within a wider function space.

However, another point of view is to count the number of free parameters of the model. In that case, model 1 would perform better than model 3, with, since the former has $2d$ parameters (the mean of each class) and the latter only has $d+1$, giving the optimizer more room to maximize in the case of 1.

Perhaps I should consider a combination of the two: models 1 and 3 learn a linear function of the input, but model 1 has more free parameters. Models 2 and 4 learn a quadratic function, although the number of free parameters is in general larger in the case of model 2.

What is the right way to approach this problem?

$\endgroup$
  • $\begingroup$ Likelihood of QDA should be higher than that of LDA, but I am not sure there exists a fixed relationship between likelihoods of LDA and of logistic regression. It might be that it's dataset-dependent. $\endgroup$ – amoeba Aug 10 '16 at 0:05
  • 1
    $\begingroup$ The relationship between LDA and QDA seems clear, as does the one between logistic regression with different sets of features, given the increased capacity of QDA and LogReg. But yes, the relationship between discriminant analysis and logistic regression is what evades me. $\endgroup$ – broncoAbierto Aug 10 '16 at 10:06
  • $\begingroup$ The exercise does consider the possibility that it is data set dependent, mind you. I've edited to add the pairs of models that the exercise asks about specifically. $\endgroup$ – broncoAbierto Aug 10 '16 at 10:07
0
$\begingroup$

Since it's a self-study question I'll give you a hint to one of the sub questions: LDA has a special property when all classes discriminated between have the exact same covariance matrix. That property should manifest itself on the training data, given the training data actually meets the assumptions, but not necessarily on the test data.

$\endgroup$
  • 1
    $\begingroup$ Sorry, but I don't get this. Why should the training data meet our assumptions? Also, the exercise asks for generalized assessments, so I don't see a reason to assume that all classes meet the LDA assumptions. If that were the case, then QDA and LDA would attain the same likelihood and would converge at the same covariance matrix, but the generalized assessment would be the same. $\endgroup$ – broncoAbierto Aug 10 '16 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.