Partial Least Squares Multinomial Regression

Question

Background:

PLS regression is a nice method to develop prediction models from data with large dimensional highly correlated measurements/predictors (e.g., spectral or other frequency domain data with a smooth envelop). Very similar to PCA regression except the orthogonal latent factors are optimized to explain variability in Y as opposed to X.

From sklearn documentation

T: x_scores_ U: y_scores_ W: x_weights_ C: y_weights_ P: x_loadings_ Q: y_loadings__ Are computed such that:

X = T P.T + Err and Y = U Q.T + Err T[:, k] = Xk W[:, k] for k in range(n_components) U[:, k] = Yk C[:, k] for k in range(n_components) x_rotations_ = W (P.T W)^(-1) y_rotations_ = C (Q.T C)^(-1) where Xk and Yk are residual matrices at iteration k.

Slides explaining PLS

For each component k, find weights u, v that optimizes: max corr(Xk u, Yk v) * std(Xk u) std(Yk u), such that |u| = 1

A way to deal with categorical variables is called PLS-DA (discriminant analysis) and approaches it as following (taken from JMP documentation):

If there are k levels, each level is represented by an indicator variable with the value 1 for rows in that level and 0 otherwise. The resulting k indicator variables are treated as continuous and the PLS analysis proceeds as it would with continuous Ys.

This is really not ideal though for multiple reasons (it can produce values < 0 and > 1, cross entropy loss is more appropriate, etc)

Question:

I wish to implement something like this except using softmax on the output for classification on multiple factors. However, how can i do this optimization? It is unclear to me how to implement this when we are no longer trying to find maximum covariance between X and Y latent factors...but instead want the orthogonal X latent factors that minimize KL divergence for the model in predicting the Y classes.

Answer 1

I found some papers from the R package plsRglm that get me most of the way there. In those cases the method of PLS for a GLM is formed (including binary and ordinal logistic regression) and thus I just need to modify the "link" function to be softmax. From the paper:

The algorithm consists of four steps:

1. computation of the $m$ PLS components $t_h$ (h = 1;:::;m);
2. generalised linear regression of y on the m retained PLS components;
3. expression of PLS-GLR in terms of the original explanatory variables;
4. Bootstrap validation of coeKcients in the 6nal model of PLS-GLR.

Hereinafter, the 6rst and the last steps are described in details while the middle two steps are shown directly in the examples as they are trivial....

published paper with detailed explanation

R Package (updated algorithm from published paper)