Paraphrased from Understanding Machine Learning by Shalev-Shwartz:

Let $\mathbf{x}_1, \dots, \mathbf{x}_m \in \mathbb{R}^d$, $\mathbf{W}$ an $n \times d$ matrix with real entries, and $\mathbf{U}$ a $d \times n$ matrix with real entries, with $n < d$. The principal component analysis (PCA) problem entails finding $\mathbf{U}$ and $\mathbf{W}$ which minimize the loss function \begin{equation} \sum_{i=1}^{m}\|\mathbf{x}_i - \mathbf{U}\mathbf{W}\mathbf{x}_i\|^{2}_{2}\text{.} \end{equation}

What is the corresponding objective funtion for partial least squares (PLS), if there is one? From what I'm getting from Introduction to Statistical Learning (ISL) by James et al. (anyone who's read this book would know it doesn't go into much mathematical detail), my guess would be if $y_1, \dots, y_m\in \mathbb{R}$, the objective function is finding the $n \times d$ matrix $\mathbf{V}$ and the $n \times 1$ matrix $\boldsymbol\beta$ which minimize
$$\sum_{i=1}^{m}\left[y_i-(\mathbf{V}\mathbf{x}_i)^{T}\boldsymbol\beta\right]^{2}$$ since ISL states:

Like PCR, PLS is a dimension reduction method, which first identifies a new set of features $Z_1, \dots, Z_M$ $(M$ corresponding to $n$ in this case) that are linear combinations of the original features, and then fits a linear model via least squares using these $M$ new features.

Am I interpreting this correctly?

  • $\begingroup$ In your formulation, what stops you from reducing it to a typical regression problem by defining $\gamma \equiv V^T\beta$? $\endgroup$ Commented Nov 7, 2017 at 15:28
  • 1
    $\begingroup$ @eric_kernfeld Ah, that's an excellent point. And I'm completely with you, PLS is extremely confusing. It took me going through 4 books to understand the theory behind PCA, and I can't find any mention of PLS other than in the Stanford texts (ISL, Elements of Statistical Learning). $\endgroup$ Commented Nov 7, 2017 at 15:32
  • $\begingroup$ I'm going to list some references to PLS questions here on CV. Have you looked at these? Q1 Q2 Q3 Q4 $\endgroup$ Commented Nov 7, 2017 at 15:55
  • 1
    $\begingroup$ Also stats.stackexchange.com/questions/179733 $\endgroup$
    – amoeba
    Commented Nov 7, 2017 at 16:01


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.