# What is the "partial" in partial least squares methods?

In partial least squares regression (PLSR) or partial least squares structural equation modelling (PLS-SEM), what does the term "partial" refer to?

• Note that Wold Jr. thinks the name "partial least square" is misleading and should have been called "projection onto latent spaces".
– Momo
Jan 29, 2015 at 19:26
• @Momo: Yes, I have read about that. However, even if PLS is misleading to a degree, the "projection onto latent spaces" is even less clear, not to mention the lack of convenience in using the term in written form. Jan 29, 2015 at 19:53

I would like to answer this question, largely based on the historical perspective, which is quite interesting. Herman Wold, who invented partial least squares (PLS) approach, hasn't started using term PLS (or even mentioning term partial) right away. During the initial period (1966-1969), he referred to this approach as NILES - abbreviation of the term and title of his initial paper on this topic Nonlinear Estimation by Iterative Least Squares Procedures, published in 1966.

As we can see, procedures that later will be called partial, have been referred to as iterative, focusing on the iterative nature of the procedure of estimating weights and latent variables (LVs). The "least squares" term comes from using ordinary least squares (OLS) regression to estimate other unknown parameters of a model (Wold, 1980). It seems that the term "partial" has its roots in the NILES procedures, which implemented "the idea of split the parameters of a model into subsets so they can be estimated in parts" (Sanchez, 2013, p. 216; emphasis mine).

The first use of the term PLS has occurred in the paper Nonlinear iterative partial least squares (NIPALS) estimation procedures, which publication marks next period of PLS history - the NIPALS modeling period. 1970s and 1980s become the soft modeling period, when, influenced by Karl Joreskog's LISREL approach to SEM, Wold transforms NIPALS approach into soft modeling, which essentially has formed the core of the modern PLS approach (the term PLS becomes mainstream in the end of 1970s). 1990s, the next period in PLS history, which Sanchez (2013) calls "gap" period, is marked largely by decreasing of its use. Fortunately, starting from 2000s (consolidation period), PLS enjoyed its return as a very popular approach to SEM analysis, especially in social sciences.

UPDATE (in response to amoeba's comment):

• Perhaps, Sanchez's wording is not ideal in the phrase that I've cited. I think that "estimated in parts" applies to latent blocks of variables. Wold (1980) describes the concept in detail.
• You're right that NIPALS was originally developed for PCA. The confusion stems from the fact that there exist both linear PLS and nonlinear PLS approaches. I think that Rosipal (2011) explains the differences very well (at least, this is the best explanation that I've seen so far).

UPDATE 2 (further clarification):

In response to concerns, expressed in amoeba's answer, I'd like to clarify some things. It seems to me that we need to distinguish the use of the word "partial" between NIPALS and PLS. That creates two separate questions about 1) the meaning of "partial" in NIPALS and 2) the meaning of "partial" in PLS (that's the original question by Phil2014). While I'm not sure about the former, I can offer further clarification about the latter.

According to Wold, Sjöström and Eriksson (2001),

The "partial" in PLS indicates that this is a partial regression, since ...

In other words, "partial" stems from the fact that data decomposition by NIPALS algorithm for PLS may not include all components, hence "partial". I suspect that the same reason applies to NIPALS in general, if it's possible to use the algorithm on "partial" data. That would explain "P" in NIPALS.

In terms of using the word "nonlinear" in NIPALS definition (do not confuse with nonlinear PLS, which represents nonlinear variant of the PLS approach!), I think that it refers not to the algorithm itself, but to nonlinear models, which can be analyzed, using linear regression-based NIPALS.

UPDATE 3 (Herman Wold's explanation):

While Herman Wold's 1969 paper seems to be the earliest paper on NIPALS, I have managed to find another one of the earliest papers on this topic. That is a paper by Wold (1974), where the "father" of PLS presents his rationale for using the word "partial" in NIPALS definition (p. 71):

3.1.4. NIPALS estimation: Iterative OLS. If one or more variables of the model are latent, the predictor relations involve not only unknown parameters, but also unknown variables, with the result that the estimation problem becomes nonlinear. As indicated in 3.1 (iii), NIPALS solves this problem by an iterative procedure, say with steps s = 1, 2, ... Each step s involves a finite number of OLS regressions, one for each predictor relation of the model. Each such regression gives proxy estimates for a sub-set of the unknown parameters and latent variables (hence the name partial least squares), and these proxy estimates are used in the next step of the procedure to calculate new proxy estimates.

References

Rosipal, R. (2011). Nonlinear partial least squares: An overview. In Lodhi H. and Yamanishi Y. (Eds.), Chemoinformatics and Advanced Machine Learning Perspectives: Complex Computational Methods and Collaborative Techniques, pp. 169-189. ACCM, IGI Global. Retrieved from http://aiolos.um.savba.sk/~roman/Papers/npls_book11.pdf

Sanchez, G. (2013). PLS path modeling with R. Berkeley, CA: Trowchez Editions. Retrieved from http://gastonsanchez.com/PLS_Path_Modeling_with_R.pdf

Wold, H. (1974). Causal flows with latent variables: Partings of the ways in the light of NIPALS modelling. European Economic Review, 5, 67-86. North Holland Publishing.

Wold, H. (1980). Model construction and evaluation when theoretical knowledge is scarce: Theory and applications of partial least squares. In J. Kmenta and J. B. Ramsey (Eds.), Evaluation of econometric models, pp. 47-74. New York: Academic Press. Retrieved from http://www.nber.org/chapters/c11693

Wold, S., Sjöström, M., & Eriksson, L. (2001). PLS-regression: A basic tool of chemometrics. Chemometrics and Intelligent Laboratory Systems, 58, 109-130. doi:10.1016/S0169-7439(01)00155-1 Retrieved from http://www.libpls.net/publication/PLS_basic_2001.pdf

• @amoeba: I believe that this paper explains PLS in contrast with other approaches in a more technical way, which you've recently discussed. However, note that the above explanation is focused on PLS regression, whereas PLS include multiple classes of system analysis (see slide 10 in the following presentation). Technical notes on slides 25-29 are IMHO also helpful. The presentation: plsmodeling.com/pls/pls-introduction. Jan 29, 2015 at 19:42
• @ Aleksandr Blekh: These are very nice references.
– Alph
Jan 29, 2015 at 21:12
• Wow, people give names to the periods of PLS history! Impressive. Jan 29, 2015 at 23:01
• Seriously though, I looked into Sanchez'es book, but still don't understand what NIPALS has to do with "the idea of split the parameters of a model into subsets so they can be estimated in parts". NIPALS was originally suggested as a method to compute principal components, right? It's quite simple. I don't see any "splits" of the parameters into "subsets" there, so I have no idea what Sanchez is talking about here. By the way, neither do I understand "nonlinear" in NIPALS. Certainly PCA is a linear technique! Jan 29, 2015 at 23:42
• @amoeba: Please see my update in response to your comment. Hope it helps. Jan 30, 2015 at 12:40

In modern expositions of PLS there is nothing "partial": PLS looks for linear combinations among variables in $X$ and among variables in $Y$ that have maximal covariance. It is an easy eigenvector problem. That's it. See The Elements of Statistical Learning, Section 3.5.2, or e.g. Rosipal & Krämer, 2005, Overview and Recent Advances in Partial Least Squares.

However, historically, as @Aleksandr nicely explains (+1), PLS was introduced by Wold who used his NIPALS algorithm to implement it; NIPALS stands for "nonlinear iterated partial least squares", so obviously P in PLS just got there from NIPALS.

Moreover, NIPALS (as I remember reading elsewhere) was not initially developed for PLS; it was introduced for PCA. Now, NIPALS for PCA is a very simple algorithm. I can present it right here. Let $\newcommand{\X}{\mathbf X}\X$ be a centered data matrix with observation in rows. The goal is to find the first principal axis $\newcommand{\v}{\mathbf v}\v$ (eigenvector of the covariance matrix) and the first principal component $\newcommand{\p}{\mathbf p}\p$ (projection of the data onto $\v$). We initialize $\p$ randomly and then iterate the following steps until convergence:

1. $\v = \X^\top \p (\p^\top \p)^{-1}$
2. Set $\|\v\|$ to $1$.
3. $\p = \X \v (\v^\top \v)^{-1}$

That's it! So the real question is why did Wold call this algorithm "partial"? The answer (as I finally understood after @Aleksandr made his third update) is that Wold viewed $\v$ and $\p$ as two [sets of] parameters, together modeling the data matrix $\X$. The algorithm updates these parameters sequentially (steps #1 and #3), i.e. it updates only one part of the parameters at a time! Hence "partial".

(Why he called it "nonlinear" I still don't understand though.)

This term is remarkably misleading, because if this is "partial" then every expectation-maximization algorithm is "partial" too (in fact, NIPALS can be seen as a primitive form of EM, see Roweis 1998). I think PLS is a good candidate for The Most Misleading Term in Machine Learning contest. Alas, it is unlikely to change, despite the efforts of Wold Jr. (see @Momo's comment above).

• You may be interested in my answer's UPDATE 2 with further clarification. Jan 30, 2015 at 14:37
• Thanks for keeping up this discussion (to prevent any misunderstandings, I should say that I did not try to criticize you in any way!). Now, to your Update2. Why do you think we should distinguish the meaning of "partial" in PLS and NIPALS? This sounds strange; PLS grew out of the work on NIPALS and this suggests that its name is simply a shortened "niPaLS". This seems to be confirmed by the Wold et al. 2001 paper that you found: "This included a simple but efficient way to estimate the parameters in these models called NIPALS [...]. This led, in turn, to the acronym PLS for these models". Jan 30, 2015 at 14:47
• Nice find! But I don't think that 1974 is "the earliest paper on NIPALS": there is a 1969 paper with NIPALS in the title (see my previous comment). Nevertheless, this quote actually sheds some light on the question: if we discuss my example of NIPALS for PCA, then Wold takes $\mathbf v$ and $\mathbf p$ as two parameters describing $\mathbf X$ and the term "partial" refers to each parameter being updated separately, i.e. only one part of parameters is updated at a time! Is it also how you read it? Jan 30, 2015 at 16:45
• Great! I think the question has finally been answered satisfactorily. And I have finally upvoted your answer, +1 :-) I edited my answer to incorporate that this new understanding. Regarding your answer: when you explained the word "partial" in Update 1 and Update 2, did you really mean the same thing as we now agreed upon? To me it looks like your answer currently contain several different interpretations... Jan 30, 2015 at 17:18
• I don't know! Perhaps it is correct. Can you elaborate on what "nonlinear models" can be analyzed using NIPALS and how? On the other hand, it is probably a completely different topic. I guess the point is that Wold developed NIPALS not to compute PCA for its own sake, but had some particular applications in mind, where he had to deal with nonlinear problems and linearized them somehow, reducing to PCA? Nowadays people present NIPALS as a simple algorithm to compute leading singular vectors, but perhaps Wold from 1969 would not agree to this view at all! Jan 30, 2015 at 17:48