4
$\begingroup$

I heard just recently about PLS-DA and I was wondering how it differs from multinomial logistic regression, since logistic regression can be also used for categorical dependent variables.

$\endgroup$
  • 1
    $\begingroup$ The question is, how are they similar in any way?? $\endgroup$ – Digio Jun 8 '17 at 18:59
1
$\begingroup$

PLS-DA is closely related to LDA: for n > p the full rank PLS-DA (i.e. using all latent variables) is the same as LDA. For 1 latent variable, PLS-DA yields the same classification as closest (Euclidean) distance in feature space. I.e. the regularization "squeezes" the pooled covariance matrix into spherical shape.

A two class problem with both classes following a (multivariate) Gaussian distribution with the same covariance matrix (i.e. the situation where LDA is optimal), both LR and LDA yield the same solution.
LR will need more samples to get to the same stability, though.

In other words, there is a somewhat indirect relationship.

There are important differences between PLS-DA and LR in how they weight cases:

  • PLS-DA (like LDA) takes all cases into account, regardless how far they are from the class boundary.
    If you (ab)use PLS for dummy regression as it is frequently done in PLS-DA (i.e. y takes class labels encoded as 0 and 1 or equivalent encodings), PLS-DA will try to "squeeze" the within class distributions to points (as required in regression).
  • LR will care mostly about cases that are close to the class boundary, classes far from the class boundary have low weight in LR.

So if you want to use PLS for classification, make sure that it is appropriate to have all cases weighting in. Two situations where this is the case are

  • the classes form nice clusters (i.e. LDA would be appropriate but you need more regularization - and for some reason don't want to do PLS-LDA)
  • the classification problem is really a regression in disguise. Example: the classes codify whether a certain metric property exceeds a threshold or not. You can then set up a proper PLS regression (with metric labels) and employ the threshold. This model will be able to benefit from cases that are at some distance to the class boundary (though you'll have to weigh this benefit against the need to get a very good prediction close to the class boundary).
  • For the latter situation, LR is also appropriate (but not LDA, nor dummy-coded PLS-DA), but it won't be able to benefit from cases further from the threshold like PLS.
$\endgroup$
-1
$\begingroup$

I think I've figured this out! I, too, have been interested in PLS-DA recently, and thanks to the answer to my question and some discussions I've had recently, I think I can answer your question! I'm going to convey my understanding in this answer, but I want to be clear that this might be wrong. Other folks here can be the judge of that.

Also, in this answer, I'm going to confine myself to the binary case, since multinomial versions of Logistic Regression and PLS-DA can be extrapolated pretty easily and it's easier to talk about the binary case.

I'm also a very visual thinker. That's why understanding this was hard for me, because looking back at the math now that I have the intuition, this is pretty clear. I'm assuming you're more like me, otherwise understanding the difference probably would have been easier for you. Okay, here goes:


Logistic regression searches for the hyperplane which best separates your data, according to some cost function (e.g. sum of squared error).

enter image description here

In what I think is a beautiful parallel, PLS-DA searches for "the opposite". It searches for the component which best explains the class variance in your dataset:

enter image description here

How does PLS-DA do this? It basically does PCA on your output space (Y), which in the case of PLS-DA is a categorical matrix of labels (or vector in the binary case). PCA, you'll recall, searches for the components with maximum variance in some dataset. So in the binary case, the component with maximum variance in the output space Y is simply the vector which "from 0 to 1" in the 1-dimensional space of Y.

Now, PLSR searches for the component in X which has maximum covariance with the first PCA component in Y, i.e. explains the most variance in your labels. How cool!

Now, perhaps the most important part of the question is: assuming the goal is feature selection, what is gained by using one or the other, if they give the same information in "opposite ways". For this, you have to consider the shape of your matrix. In the case of bioinformatics (where I work) we frequently have many features and few examples. In those cases, the problem is "ill-posed" or "ill-conditioned" and logistic regression has high variance, but PLS-DA is much more resistant to this, which is why:

PLS regression is today most widely used in chemometrics and related areas. It is also used in bioinformatics, sensometrics, neuroscience and anthropology. -- Wikipedia

I hope that answers your question! I hope that's correct too ;)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.