PLS regression is a nice method to develop prediction models from data with large dimensional highly correlated measurements/predictors (e.g., spectral or other frequency domain data with a smooth envelop). Very similar to PCA regression except the orthogonal latent factors are optimized to explain variability in Y as opposed to X.
T: x_scores_ U: y_scores_ W: x_weights_ C: y_weights_ P: x_loadings_ Q: y_loadings__ Are computed such that:
X = T P.T + Err and Y = U Q.T + Err T[:, k] = Xk W[:, k] for k in range(n_components) U[:, k] = Yk C[:, k] for k in range(n_components) x_rotations_ = W (P.T W)^(-1) y_rotations_ = C (Q.T C)^(-1) where Xk and Yk are residual matrices at iteration k.
Slides explaining PLS
For each component k, find weights u, v that optimizes: max corr(Xk u, Yk v) * std(Xk u) std(Yk u), such that |u| = 1
A way to deal with categorical variables is called PLS-DA (discriminant analysis) and approaches it as following (taken from JMP documentation):
If there are k levels, each level is represented by an indicator variable with the value 1 for rows in that level and 0 otherwise. The resulting k indicator variables are treated as continuous and the PLS analysis proceeds as it would with continuous Ys.
This is really not ideal though for multiple reasons (it can produce values < 0 and > 1, cross entropy loss is more appropriate, etc)
I wish to implement something like this except using softmax on the output for classification on multiple factors. However, how can i do this optimization? It is unclear to me how to implement this when we are no longer trying to find maximum covariance between X and Y latent factors...but instead want the orthogonal X latent factors that minimize KL divergence for the model in predicting the Y classes.