4
$\begingroup$

We know one of the definition of partial least squares (PLS) is:

$$\max\limits_{\alpha_x,\alpha_y}Cov(\alpha_x^Tx,\alpha^T_yy)$$ $$||\alpha_x|| = ||\alpha_y|| = 1.$$

Here $x = (x_1,\cdots,x_n)$ is the dependent variable; $y = (y_1,\cdots,y_m)$ is the responding variable.

We consider a special case $\dim y = m = 1.$ Then $\alpha_y = 1.$ Therefore:

$$Cov(\alpha_x^Tx,\alpha^T_yy) = Cov(\alpha_x^Tx,y) = Cov(x,y)\alpha_x = \Big(Cov(x_1,y),\cdots,Cov(x_n,y)\Big)\alpha_x.$$ since $||\alpha_x|| = 1,$ then

$$\max Cov(\alpha_x^Tx,y) = max \left\{Cov(x_1,y),\cdots,Cov(x_n,y)\right\} = Cov(x_k,y).$$ Where $\alpha_x^k = 1, \alpha_x^i = 0$ for $i \neq k.$ Namely,

For the 1 dimensional responding variable, the first component of PLS is the component of $x$ with the largest covariances (absolute value) with $y.$

Am I correct? The conclusion seems a little bit strange.

$\endgroup$

1 Answer 1

2
$\begingroup$

Well the problem in your math comes from the equation you are providing as solution for the PLS algorithm. Let me write down the math on how to find this equation. I will start by the NIPALS algorithm for PLS2:


Given two matrices $X\in\mathbb{R}^{n\times p}$ and $Y\in\mathbb{R}^{n\times m}$, pick some $u=Y_j$ a column vector of $Y$ and iterate:

  1. $w = \dfrac{X^\prime u}{u^\prime u}$; $w = \dfrac{w}{\|w\|^2}$. Or in equivalent step: $w = \dfrac{X^\prime u}{\|X^\prime u\|}$

  2. $t =Xw$

  3. $c = \dfrac{Y^\prime t}{t^\prime t}$; $c = \dfrac{c}{\|c\|^2}$. Or in equivalent step: $c = \dfrac{Y^\prime c}{\|Y^\prime c\|}$

  4. $u = Yc$

Repeat the previous steps until the difference in $w$ from one iteration to the next is small.


The maximization problem of PLS2

Now based on the algorithm itself, we can easily derive the maximization problem that PLS2 is solving. We just need to use the different equations for $u$, $t$ and $c$ and substitute:

$$w = \dfrac{X^\prime u}{\|X^\prime u\|} = \dfrac{X^\prime Yc}{\|X^\prime Yc\|} = \dfrac{X^\prime YY^\prime t \frac{1}{\|Y^\prime t\|}}{\|X^\prime YY^\prime t \frac{1}{\|Y^\prime t\|}\|} = \dfrac{X^\prime YY^\prime t}{\|X^\prime YY^\prime t\|} = \dfrac{X^\prime YY^\prime Xw}{\|X^\prime YY^\prime Xw\|}$$

Now if we observe that $\|X^\prime YY^\prime Xw\|$ is a scalar and rename it as $\lambda$ we see the equation:

$$w = \frac{1}{\lambda}X^\prime YY^\prime Xw \Rightarrow (X^\prime YY^\prime X)w = \lambda w$$

Now we see two things:

  1. $(X^\prime YY^\prime X) = (Y^\prime X)^\prime (Y^\prime X)$, and if we assume that $X$ and $Y$ are both centered, then $cov(X, Y)=Y^\prime X$, so,

$$ (X^\prime YY^\prime X) = cov(X, Y)^\prime cov(X, Y)$$

  1. If we go back to our linear algebra notes, (the part on eigenvectors) we see that $w$ must be an eigenvector of $cov(X, Y)^\prime cov(X, Y)$

nd finally, the last piece of linear algebra that we need: the spectral theorem tells us that given a symetric real matrix $A$, the solution to the optimization problem

$$max\{w^\prime A w\} \quad s.t. \quad \|w\|=1$$

is the largest eigenvector of $A$. We see that $cov(X, Y)^\prime cov(X, Y)$ is a real symetric matrix, so we apply here the spectral theorem and we discover that the vector $w$ that we were seeking was the solution to the optimization problem

$$max\{w^\prime cov(X, Y)^\prime cov(X,Y) w\} \quad s.t. \quad \|w\|=1$$ And that $w$ is then the eigenvector associated to the largest eigenvalue of $cov(X, Y)^\prime cov(X,Y)$

Actually, the NIPALS algorithm can be seen simply as the power method, a numerical algorithm for computing the eigenvectors of a matrix. But in any case, here is the optimization problem that the PLS is actually solving. As you see, in your equations you were missing that the covariance was squared.

However, in PLS1 there is still a relation between the first pls component and the covariance between X and Y. If we go back to the NIPALS

  1. $w = \dfrac{X^\prime u}{\|X^\prime u\|}$. If $Y$ is one-dimensional, then $u=y$ and then $w = \dfrac{X^\prime y}{\|X^\prime y\|} = \dfrac{cov(X, y)}{\|cov(X, y)\|}$

  2. $t =Xw$

  3. $c = \dfrac{Y^\prime c}{\|Y^\prime c\|}$. If $Y$ is one-dimensional, this simplifies to $c=1$

  4. $u = Yc$. If $Y$ is one-dimensional, this simplifies (again) to $u=y$

So, in the PLS1 algorithm, when $Y$ is one-dimensional, the first PLS component $w$ is simply the normalized covariance between $X$ and $Y$

$\endgroup$
7
  • $\begingroup$ sorry, I know where I am incorrect, but one thing I don't understand that How do you get this objective function $\max\{w'cov(X,y)'cov(X,y)w\}$ linked to PLS, since in the NIPALS, we cannot see it is an optimization problem. But I guess it may be the same story for the simply case of NIPALS in PCA. Or equivalently, from the algorithm, we only know $w$ is a eign vector, why is it the largest one? $\endgroup$ Jun 27, 2021 at 9:28
  • $\begingroup$ I am afraid I do not understand your question. In my answer I show you how the NIPALS can be seen as a covariance maximization problem, if you can be more specific on your doubts I will gladly answer. $\endgroup$ Jun 27, 2021 at 15:58
  • $\begingroup$ Let me clarify, the 4 steps of NIPALS only show that $w$ is the eigen vector of matrix $X'YY'X.$ However, how do you guarantee $w$ is the first component i.e. corresponding the largest eigen value? $\endgroup$ Jun 27, 2021 at 16:13
  • $\begingroup$ That is as in PCA and comes from the fact that we are trying to maximize a quadratic form of a real valued symetric function. The spectral theorem tells us that exist an orthonormal basis (made of its eigenvectors) that allow you to decompose the matrix into $cov(X,Y)^\prime cov(X,Y)$ = $UDU^\prime$. Then, using the fact that the columns of $U$ are orthogonal, you can prove that the value for $w$ that maximize the problem defined above must be that of the eigenvector linked to the largest eigenvalue. $\endgroup$ Jun 28, 2021 at 7:02
  • $\begingroup$ I agree that $w$ maximizing the quadratic form is equivalent to the largest eigen value. So you mean the algorithm of NIPALS itself already guarantee the largest eigen value (independent on the choice of initial value) and the prove is not trivial? $\endgroup$ Jun 28, 2021 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.