one dimension PLS

Question

We know one of the definition of partial least squares (PLS) is:

$$\max\limits_{\alpha_x,\alpha_y}Cov(\alpha_x^Tx,\alpha^T_yy)$$ $$||\alpha_x|| = ||\alpha_y|| = 1.$$

Here $x = (x_1,\cdots,x_n)$ is the dependent variable; $y = (y_1,\cdots,y_m)$ is the responding variable.

We consider a special case $\dim y = m = 1.$ Then $\alpha_y = 1.$ Therefore:

$$Cov(\alpha_x^Tx,\alpha^T_yy) = Cov(\alpha_x^Tx,y) = Cov(x,y)\alpha_x = \Big(Cov(x_1,y),\cdots,Cov(x_n,y)\Big)\alpha_x.$$ since $||\alpha_x|| = 1,$ then

$$\max Cov(\alpha_x^Tx,y) = max \left\{Cov(x_1,y),\cdots,Cov(x_n,y)\right\} = Cov(x_k,y).$$ Where $\alpha_x^k = 1, \alpha_x^i = 0$ for $i \neq k.$ Namely,

For the 1 dimensional responding variable, the first component of PLS is the component of $x$ with the largest covariances (absolute value) with $y.$

Am I correct? The conclusion seems a little bit strange.

Answer 1

Well the problem in your math comes from the equation you are providing as solution for the PLS algorithm. Let me write down the math on how to find this equation. I will start by the NIPALS algorithm for PLS2:

---

Given two matrices $X\in\mathbb{R}^{n\times p}$ and $Y\in\mathbb{R}^{n\times m}$, pick some $u=Y_j$ a column vector of $Y$ and iterate:

1. $w = \dfrac{X^\prime u}{u^\prime u}$; $w = \dfrac{w}{\|w\|^2}$. Or in equivalent step: $w = \dfrac{X^\prime u}{\|X^\prime u\|}$

2. $t =Xw$

3. $c = \dfrac{Y^\prime t}{t^\prime t}$; $c = \dfrac{c}{\|c\|^2}$. Or in equivalent step: $c = \dfrac{Y^\prime c}{\|Y^\prime c\|}$

4. $u = Yc$

Repeat the previous steps until the difference in $w$ from one iteration to the next is small.

---

The maximization problem of PLS2

Now based on the algorithm itself, we can easily derive the maximization problem that PLS2 is solving. We just need to use the different equations for $u$, $t$ and $c$ and substitute:

$$w = \dfrac{X^\prime u}{\|X^\prime u\|} = \dfrac{X^\prime Yc}{\|X^\prime Yc\|} = \dfrac{X^\prime YY^\prime t \frac{1}{\|Y^\prime t\|}}{\|X^\prime YY^\prime t \frac{1}{\|Y^\prime t\|}\|} = \dfrac{X^\prime YY^\prime t}{\|X^\prime YY^\prime t\|} = \dfrac{X^\prime YY^\prime Xw}{\|X^\prime YY^\prime Xw\|}$$

Now if we observe that $\|X^\prime YY^\prime Xw\|$ is a scalar and rename it as $\lambda$ we see the equation:

$$w = \frac{1}{\lambda}X^\prime YY^\prime Xw \Rightarrow (X^\prime YY^\prime X)w = \lambda w$$

Now we see two things:

1. $(X^\prime YY^\prime X) = (Y^\prime X)^\prime (Y^\prime X)$, and if we assume that $X$ and $Y$ are both centered, then $cov(X, Y)=Y^\prime X$, so,

$$ (X^\prime YY^\prime X) = cov(X, Y)^\prime cov(X, Y)$$

2. If we go back to our linear algebra notes, (the part on eigenvectors) we see that $w$ must be an eigenvector of $cov(X, Y)^\prime cov(X, Y)$

nd finally, the last piece of linear algebra that we need: the spectral theorem tells us that given a symetric real matrix $A$, the solution to the optimization problem

$$max\{w^\prime A w\} \quad s.t. \quad \|w\|=1$$

is the largest eigenvector of $A$. We see that $cov(X, Y)^\prime cov(X, Y)$ is a real symetric matrix, so we apply here the spectral theorem and we discover that the vector $w$ that we were seeking was the solution to the optimization problem

$$max\{w^\prime cov(X, Y)^\prime cov(X,Y) w\} \quad s.t. \quad \|w\|=1$$ And that $w$ is then the eigenvector associated to the largest eigenvalue of $cov(X, Y)^\prime cov(X,Y)$

Actually, the NIPALS algorithm can be seen simply as the power method, a numerical algorithm for computing the eigenvectors of a matrix. But in any case, here is the optimization problem that the PLS is actually solving. As you see, in your equations you were missing that the covariance was squared.

However, in PLS1 there is still a relation between the first pls component and the covariance between X and Y. If we go back to the NIPALS

1. $w = \dfrac{X^\prime u}{\|X^\prime u\|}$. If $Y$ is one-dimensional, then $u=y$ and then $w = \dfrac{X^\prime y}{\|X^\prime y\|} = \dfrac{cov(X, y)}{\|cov(X, y)\|}$

2. $t =Xw$

3. $c = \dfrac{Y^\prime c}{\|Y^\prime c\|}$. If $Y$ is one-dimensional, this simplifies to $c=1$

4. $u = Yc$. If $Y$ is one-dimensional, this simplifies (again) to $u=y$

So, in the PLS1 algorithm, when $Y$ is one-dimensional, the first PLS component $w$ is simply the normalized covariance between $X$ and $Y$