# What is the relationship between PLS Regression / Discriminant Analysis (PLSR/PLS-DA) and Linear Regression

I'm interested in the problem of feature selection, and I came across PLS-DA, which seems to be a "hack" on PLSR (one more reference). PLSR relates a matrix X to a matrix Y, and PLS-DA relates a matrix X to some labels, represented as a matrix Y.

Now, in PLSR/DA, you can decide the number of components you'd like to use. to relate X to Y.

If the number of components is 1, and Y is only a vector of 0/1 class labels, does PLS-DA just become Linear (OLS) regression? If not, how is it different?

PLS-DA is in fact a hack on PLSR. PLS tries to find uncorrelated projection (scores) of X that maximizes covariance between X and Y. If you are using all components (called latent variables for PLS) then it becomes OLS. By using a single component or any number of components that is less the maximum number possible, you are actually penalizing the rest of the directions which are assumed to be less useful for regression (such as irrevelant information in X about Y).

PLS with SIMPLS algorithm produces:

• m: number of observations
• n: number of variables
• l: number of classes
• a: number of components(LVs)

$\mathbf T_(mxa) = \mathbf X_(mxn) \mathbf R(nxa)$ where T is your X scores for a components, R is the weights for a components

$\mathbf {\hat Y_{(mxl)}} = \mathbf T_(mxa) \mathbf Q_(lxa)'$ This is the prediction step.

Therefore you can define the regression coefficents:

$\mathbf B_(nxl) = \mathbf R(nxa) \mathbf Q_{(lxa)}'$

So the prediction can be done with a single regression matrix B

$\mathbf {\hat Y_{(mxl)}} = \mathbf X_(mxn) \mathbf B_(nxl)$

Now if you retain all components ( by setting $a = min(m-1,n-1)$ ) then B will be equivalent of solving OLS for $Y = XB$ but if you choose smaller number of components it is now PLSR and in that case the only thing that remains common between OLS and PLSR is the size of the B matrix.

For more details, here is the original article: De Jong, Sijmen. "SIMPLS: an alternative approach to partial least squares regression." Chemometrics and intelligent laboratory systems 18, no. 3 (1993): 251-263.

• "PLS tries to find orthogonal directions for X" - "orthogonal" here might need some clarification. Are the axes (weights) orthogonal? Or are the projections (scores) uncorrelated? I think it's the latter, but you sentence sounds as if it's the former. – amoeba Jun 17 '17 at 11:24
• You are right, only the scores are uncorrelated and weights are only normalized. I have corrected that, also changed W to R to avoid confusion (NIPALS use W, whereas SIMPLS use R for weights) among different PLS papers. – theGD Jun 17 '17 at 11:41
• Note that the "hack" is a very generic one: it can be (ab)used to apply any sort of regression algorithm to a classification problem. Whether (or when) this approach is sensible, is a totally different question. – cbeleites unhappy with SX Jan 27 '18 at 12:45