# Partial Least Squares Multinomial Regression

Background:

PLS regression is a nice method to develop prediction models from data with large dimensional highly correlated measurements/predictors (e.g., spectral or other frequency domain data with a smooth envelop). Very similar to PCA regression except the orthogonal latent factors are optimized to explain variability in Y as opposed to X.

X = T P.T + Err and Y = U Q.T + Err T[:, k] = Xk W[:, k] for k in range(n_components) U[:, k] = Yk C[:, k] for k in range(n_components) x_rotations_ = W (P.T W)^(-1) y_rotations_ = C (Q.T C)^(-1) where Xk and Yk are residual matrices at iteration k.

Slides explaining PLS

For each component k, find weights u, v that optimizes: max corr(Xk u, Yk v) * std(Xk u) std(Yk u), such that |u| = 1

A way to deal with categorical variables is called PLS-DA (discriminant analysis) and approaches it as following (taken from JMP documentation):

If there are k levels, each level is represented by an indicator variable with the value 1 for rows in that level and 0 otherwise. The resulting k indicator variables are treated as continuous and the PLS analysis proceeds as it would with continuous Ys.

This is really not ideal though for multiple reasons (it can produce values < 0 and > 1, cross entropy loss is more appropriate, etc)

Question:

I wish to implement something like this except using softmax on the output for classification on multiple factors. However, how can i do this optimization? It is unclear to me how to implement this when we are no longer trying to find maximum covariance between X and Y latent factors...but instead want the orthogonal X latent factors that minimize KL divergence for the model in predicting the Y classes.

• The variant of PLS that actually maximizes the covariance is SIMPLS. In the article, the author explains the derivation thus you can go through it to construct your own method with different criterias: sciencedirect.com/science/article/abs/pii/016974399385002X – theGD Apr 23 '18 at 18:37
• Thanks -- but I guess I was hoping for a suggestion that gets me further along than this approach (which was my last resort). If i have to go so far as to construct my own then it almost seems like I could publish it?? – JPJ Apr 23 '18 at 23:59

I found some papers from the R package plsRglm that get me most of the way there. In those cases the method of PLS for a GLM is formed (including binary and ordinal logistic regression) and thus I just need to modify the "link" function to be softmax. From the paper:

The algorithm consists of four steps:

1. computation of the $m$ PLS components $t_h$ (h = 1;:::;m);
2. generalised linear regression of y on the m retained PLS components;
3. expression of PLS-GLR in terms of the original explanatory variables;
4. Bootstrap validation of coeKcients in the 6nal model of PLS-GLR.

Hereinafter, the 6rst and the last steps are described in details while the middle two steps are shown directly in the examples as they are trivial....

published paper with detailed explanation

R Package (updated algorithm from published paper)

• Thank you for the question and the answer. It provided me a new way of thinking about PLS, Somehow, I thought PLS was only about maximizing covariance, never thought about other options. – theGD May 7 '18 at 10:46