I heard just recently about PLS-DA and I was wondering how it differs from multinomial logistic regression, since logistic regression can be also used for categorical dependent variables.

  • $\begingroup$ The question is, how are they similar in any way?? $\endgroup$ – Digio Jun 8 '17 at 18:59

I think I've figured this out! I, too, have been interested in PLS-DA recently, and thanks to the answer to my question and some discussions I've had recently, I think I can answer your question! I'm going to convey my understanding in this answer, but I want to be clear that this might be wrong. Other folks here can be the judge of that.

Also, in this answer, I'm going to confine myself to the binary case, since multinomial versions of Logistic Regression and PLS-DA can be extrapolated pretty easily and it's easier to talk about the binary case.

I'm also a very visual thinker. That's why understanding this was hard for me, because looking back at the math now that I have the intuition, this is pretty clear. I'm assuming you're more like me, otherwise understanding the difference probably would have been easier for you. Okay, here goes:

Logistic regression searches for the hyperplane which best separates your data, according to some cost function (e.g. sum of squared error).

enter image description here

In what I think is a beautiful parallel, PLS-DA searches for "the opposite". It searches for the component which best explains the class variance in your dataset:

enter image description here

How does PLS-DA do this? It basically does PCA on your output space (Y), which in the case of PLS-DA is a categorical matrix of labels (or vector in the binary case). PCA, you'll recall, searches for the components with maximum variance in some dataset. So in the binary case, the component with maximum variance in the output space Y is simply the vector which "from 0 to 1" in the 1-dimensional space of Y.

Now, PLSR searches for the component in X which has maximum covariance with the first PCA component in Y, i.e. explains the most variance in your labels. How cool!

Now, perhaps the most important part of the question is: assuming the goal is feature selection, what is gained by using one or the other, if they give the same information in "opposite ways". For this, you have to consider the shape of your matrix. In the case of bioinformatics (where I work) we frequently have many features and few examples. In those cases, the problem is "ill-posed" or "ill-conditioned" and logistic regression has high variance, but PLS-DA is much more resistant to this, which is why:

PLS regression is today most widely used in chemometrics and related areas. It is also used in bioinformatics, sensometrics, neuroscience and anthropology. -- Wikipedia

I hope that answers your question! I hope that's correct too ;)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.