A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.
38

Section 3.5.2 in The Elements of Statistical Learning is useful because it puts PLS regression in the right context (of other regularization methods), but is indeed very brief, and leaves some important statements as exercises. In addition, it only considers a case of a univariate dependent variable $\mathbf y$. The literature on PLS is vast, but can be ...


30

I think there is no single answer to your question - it depends upon many situation, data and what you are trying to do. Some of the modification can be or should be modified to achieve the goal. However the following general discussion can help. Before jumping to into the more advanced methods let's discussion of basic model first: Least Squares (LS) ...


27

Tijl De Bie wrote an interesting chapter "Eigenproblems in Pattern Recognition" which talks about exactly these from a primal/dual perspective. The three tables at the end summarise really nicely from an optimisation perspective:


17

PLS regression relies on iterative algorithms (e.g., NIPALS, SIMPLS). Your description of the main ideas is correct: we seek one (PLS1, one response variable/multiple predictors) or two (PLS2, with different modes, multiple response variables/multiple predictors) vector(s) of weights, $u$ (and $v$), say, to form linear combination(s) of the original variable(...


17

I would like to answer this question, largely based on the historical perspective, which is quite interesting. Herman Wold, who invented partial least squares (PLS) approach, hasn't started using term PLS (or even mentioning term partial) right away. During the initial period (1966-1969), he referred to this approach as NILES - abbreviation of the term and ...


17

A geometrical interpretation The estimator described in the question is the Lagrange multiplier equivalent of the following optimization problem: $$\text{minimize $f(\beta)$ subject to $g(\beta) \leq t$ and $h(\beta) = 1$ } $$ $$\begin{align} f(\beta) &= \lVert y-X\beta \lVert^2 \\ g(\beta) &= \lVert \beta \lVert^2\\ h(\beta) &= \lVert ...


15

These are three different methods, and none of them can be seen as a special case of another. Formally, if $\mathbf X$ and $\mathbf Y$ are centered predictor ($n \times p$) and response ($n\times q$) datasets and if we look for the first pair of axes, $\mathbf w \in \mathbb R^p$ for $\mathbf X$ and $\mathbf v \in \mathbb R^q$ for $\mathbf Y$, then these ...


13

Warning: R uses the term "loadings" in a confusing way. I explain it below. Consider dataset $\mathbf{X}$ with (centered) variables in columns and $N$ data points in rows. Performing PCA of this dataset amounts to singular value decomposition $\mathbf{X} = \mathbf{U} \mathbf{S} \mathbf{V}^\top$. Columns of $\mathbf{US}$ are principal components (PC "scores")...


11

The sum of variances of all PLS components is normally less than 100%. There are many variants of partial least squares (PLS). What you used here, is PLS regression of a univariate response variable $\mathbf y$ onto several variables $\mathbf X$; this algorithm is traditionally known as PLS1 (as opposed to other variants, see Rosipal & Kramer, 2006, ...


10

Say your predictor matrix is $X$ and your response vector is $y$. PCA is concerned only with the (co)variance within the predictor matrix $X$ itself, while a regression model is (also) concerned with the covariance between $X$ and the response $y$. If there is no relationship between these concepts, dimension reduction by PCA can be harmful to your ...


10

This is an algebraic counterpart to @Martijn's beautiful geometric answer. First of all, the limit of $$\hat{\boldsymbol\beta}_\lambda^* = \arg\min\Big\{\|\mathbf y - \mathbf X \boldsymbol \beta\|^2+\lambda\|\boldsymbol\beta\|^2\Big\} \:\:\text{s.t.}\:\: \|\mathbf X \boldsymbol\beta\|^2=1$$ when $\lambda\to\infty$ is very simple to obtain: in the limit, the ...


9

The possible confusion here, as @amoeba points out in a comment, is the difference between variable selection and dimensionality reduction. Both PCA and PLS are intended to reduce the dimensionality of the problem. If you have measured 8 variables on each of your cases (and you have more than 8 cases) then the original dimension is 8. PCA and PLS help you ...


8

They are different methods, independently of the number of response variables. Both methods combine PCA with ordinary multiple regression but it's done in a crucially different way. For a matrix of predictor variables X and one of dependent variables Y, principal component regression performs a PCA on predictor matrix X and then uses those principal ...


7

In modern expositions of PLS there is nothing "partial": PLS looks for linear combinations among variables in $X$ and among variables in $Y$ that have maximal covariance. It is an easy eigenvector problem. That's it. See The Elements of Statistical Learning, Section 3.5.2, or e.g. Rosipal & Krämer, 2005, Overview and Recent Advances in Partial Least ...


7

The first question is what is the difference between "PLS path modeling" and "PLS regression"? None, they are synonyms. To make it more general, what are structural equation modeling (SEM), path modeling and regression? To my understanding regression focuses more on prediction while SEM focus is on the relationship between response and predictors and ...


6

Welcome to cross validated! Approach 1 Have a look at chapters 7.10 and 7.11 of The Elements of Statistical Learning. I think the basic idea is to calculate the uncertainty on the test results for the different numbers of latent variables. That gives you an idea which differences you cannot trust to be real differences. Do not forget that choosing the ...


6

Selecting the number of components for PLS is a bit trickier than for PCA. For instance, one reason is that quantities such as "explained variance" are more complex since you have both the $\mathbf X$ and $\mathbf Y$ parts of the model contributing to the variation explained. Thus for PLS, cross-validation tends to be the default method for selecting the ...


6

Probabilistic canonical correlation analysis (probabilistic CCA, PCCA) was introduced in Bach & Jordan, 2005, A Probabilistic Interpretation of Canonical Correlation Analysis, several years after Tipping & Bishop presented their probabilistic principal component analysis (probabilistic PCA, PPCA). Very briefly, it is based on the following ...


5

When we say that the standard OLS regression has some assumptions, we mean that these assumptions are needed to derive some desirable properties of the OLS estimator such as e.g. that it is the best linear unbiased estimator -- see Gauss-Markov theorem and an excellent answer by @mpiktas in What is a complete list of the usual assumptions for linear ...


5

For the benefit of other readers I will briefly explain what the permutation test is in this context. In this specific example there is a binary dependent variable $y$, a large number of independent variables $X$ with $n\ll p$, and a specific algorithm (PLS-DA) to predict $y$ from $X$ with one hyperparameter (number $k$ of PLS components). To find the ...


5

I'm mostly using the papers Paul Geladi and Bruce R. Kowalski: Partial least-squares regression: a tutorial, Analytica Chimica Acta, 185, 1-17 (1986). DOI: 10.1016/0003-2670(86)80028-9 and Mevik, B.-H. & Wehrens, R.: The pls Package: Principal Component and Partial Least Squares Regression in R, Journal of Statistical Software, 18, 1 - 24 (2007). DOI: ...


5

A better link to the blog on "Y-aware PCA" is here. The authors of that blog have an R package vtreat that implements this and other approaches to conditioning variables before analysis. As noted in some comments, Y-aware PCA is related to partial least squares (PLS). It weights predictor variables according to their single-variable relations to the outcome ...


5

There are mainly two algorithms for PLSR namely NIPALS and SIMPLS. SIMPLS algorithm is generally faster yet numerically less stable(in most cases the difference is very small). The original article of SIMPLS provides the steps which starts with mean centering both X and Y. The maintainer of the package probably relies on these steps. However, directly ...


5

Quick answer which I will expand in few days is PLS-DA is a supervised method where you supply the information about each sample's group. PCA, on the other hand, is an unsupervised method which means that you are just projecting the data to, lets say, 2D space in a good way to observe how the samples are clustering by theirselves. PCA, after coloring of ...


4

Each package has been developed for some specific purposes although they may all have some "Partial least squares regression" term as the their general explanation. For example Package plsRglm is designed for Partial least squares Regression for (weighted) generalized linear models and kfold crossvalidation of such models using various criteria. Package ...


4

PLS isn't necessarily a cure for insufficient sample size (though I recognize you're not claiming otherwise exactly). Check out Westland (2010) for a discussion of factors that determine the necessary sample size (and often determine that it's much larger than published articles often use). He discusses PLS too. One notable argument is that: In PLS, the ...


4

Inclusion/exclusion of variates (step 3): I understand that you ask which of the original measurement channels to include into the modeling. Is such a decision sensible for your data? E.g. I work mainly with spectroscopic data, for which PLS is frequently and successfully used. Well measured spectra have a high correlation betweeen neighbour variates and ...


4

You want your evaluation to tell you something useful about your system's performance. Using a specific, held-out test set is nice because it tells you how the system will perform on totally new data. On the other hand, it's hard-to-impossible to perform meaningful inference (i.e., "In general, is my system better than this other one?") with only a single ...


4

There is a good summary of PLS regression in Rosipal, R and Krämer, N (2006). Overview and recent advances in Partial Least Squares. In Saunder, C, Grobelnik, M, Gunn, S, and Shawe-Taylors (Eds.), Subspace, Latent Structure and Feature Selection, pp. 34-51, Springer. Using your notation, where $P$ and $Q$ are matrices of loadings, of dimensions $(...


4

Bootstrapping typically involves sampling with replacement from your sample data. The size of each bootstrapped sample should be the same as your actual sample size (i.e., n=50). This allows you to study the sampling distribution of your estimators in your actual sample size. As @Andy notes, you can arbitrarily increase your bootstrapped sample size (lets ...


Only top voted, non community-wiki answers of a minimum length are eligible