Questions tagged [partial-least-squares]

A class of linear methods for modeling the relationship between two groups of variables, X and Y. Includes PLS regression.

Filter by
Sorted by
Tagged with
38
votes
2answers
7k views

Theory behind partial least squares regression

Can anyone recommend a good exposition of the theory behind partial least squares regression (available online) for someone who understands SVD and PCA? I have looked at many sources online and have ...
30
votes
1answer
7k views

PCA, LDA, CCA, and PLS

How are PCA, LDA, CCA, and PLS related? They all seem "spectral" and linear algebraic and very well understood (say 50+ years of theory built around them). They are used for very different things (PCA ...
23
votes
2answers
1k views

The limit of “unit-variance” ridge regression estimator when $\lambda\to\infty$

Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
19
votes
1answer
3k views

What is the connection between partial least squares, reduced rank regression, and principal component regression?

Are reduced rank regression and principal component regression just special cases of partial least squares? This tutorial (Page 6, "Comparison of Objectives") states that when we do partial least ...
17
votes
1answer
10k views

Regression in $p>n$ setting: how to choose regularization method (Lasso, PLS, PCR, ridge)?

I am trying see whether to go for ridge regression, LASSO, principal component regression (PCR), or Partial Least Squares (PLS) in a situation where there are large number of variables / features ($p$)...
16
votes
2answers
3k views

What is the “partial” in partial least squares methods?

In partial least squares regression (PLSR) or partial least squares structural equation modelling (PLS-SEM), what does the term "partial" refer to?
13
votes
1answer
22k views

Partial least squares regression in R: why is PLS on standardized data not equivalent to maximizing correlation?

I am very new in partial least squares (PLS) and I try to understand the output of the R function plsr() in the pls package. Let ...
13
votes
1answer
2k views

Difference between PLS regression and PLS path modeling. Criticism of PLS

This question was asked here but no one gave a good answer. So I think it's a good idea to bring it up again and also I would like to add some more comments/questions. The first question is what is ...
13
votes
3answers
6k views

Model assumptions of partial least squares (PLS) regression

I am trying to find information regarding the assumptions of PLS regression (single $y$). I am especially interested in a comparison of the assumptions of PLS with regards to those of OLS regression. ...
11
votes
1answer
25k views

What is the difference between “loadings” and “correlation loadings” in PCA and PLS?

One common thing to do when doing Principal Component Analysis (PCA) is to plot two loadings against each other to investigate the relationships between the variables. In the paper accompanying the ...
11
votes
1answer
4k views

Why do all the PLS components together explain only a part of the variance of the original data?

I have a dataset consisting of 10 variables. I ran partial least squares (PLS) to predict a single response variable by these 10 variables, extracted 10 PLS components, and then computed the variance ...
10
votes
2answers
4k views

How to compute the confidence intervals on regression coefficients in PLS?

The underlying model of PLS is that a given $n \times m$ matrix $X$ and $n$ vector $y$ are related by $$X = T P' + E,$$ $$y = T q' + f,$$ where $T$ is a latent $n \times k$ matrix, and $E, ...
9
votes
1answer
7k views

PLS (partial least squares) weights, loadings, and scores interpretations

In SKLearn PLSRegression, several items can be called after a model is trained: Loadings Scores Weights All the above are separated by X and Y I intuitively understand that x_scores and y_scores ...
8
votes
1answer
4k views

Is each of the PCA or PLS components just one of the original variables?

I am confused about what a component is in PCA and PLS. Are the components just the original variables but not necessarily in the same order? For example, in PCA, if I had 8 variables in my data, ...
7
votes
1answer
444 views

Probabilistic models for partial least squares, reduced rank regression, and canonical correlation analysis?

This question results from the discussion following a previous question: What is the connection between partial least squares, reduced rank regression, and principal component regression? For ...
6
votes
1answer
13k views

How to use PCA in regression?

I'm currently reading in the Applied Predictive Modeling book about PCA for dimensionality reduction. I've read the following: If the predictive relationship between the predictors and response is ...
6
votes
1answer
2k views

Mean centering or not in the context of Partial Least Squares

In my current project, I'm using PLS regression on infrared spectra (FTIR). For this I'm using R and the pls function from the plsr package. ...
6
votes
1answer
8k views

Validation metrics (R2 and Q2) for Partial Least Squares (PLS) Regression

I'm attempting to validate my Partial Least Squares (PLS) -regression model. From documentation and other readings regarding PLS regression I've come to understand that there are generally two ...
6
votes
1answer
859 views

What does “orthogonalize” mean?

I have read in The Elements of Statistical Learning book and particularly in the Partial Least Squares (PLS) section: Orthogonalize each $x_j^{(m−1)}$ with respect to $z_m$. I would like to know ...
6
votes
2answers
3k views

How do you predict the value of new instance, when the training data were normalized?

I estimated a Partial Least Squares model where the X matrix had normalized columns. Now I want to predict the value for a new instance (which is a frequency vector summing to one.) I assume that if I ...
6
votes
1answer
1k views

Measuring predictive accuracy for multiple dependent variables

In machine learning and in statistics there exist plenty of measures which estimate the performance of a predictive model. For example, classification accuracy, area under ROC curve ... for ...
6
votes
0answers
394 views

What, if any, dissimilarity is preserved in partial least squares (PLS)?

When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
5
votes
1answer
447 views

How would Y-aware PCA for binaries look?

I recently stumbled upon Y-aware PCA in the blog of win-vector. They describe how PCA can be adjusted not to explain variation in $X$ but covariation of $X$ and $Y$. This is explained for the case ...
5
votes
1answer
9k views

PCA and PLS: testing variables for significance

I'm trying to understand the process for statistical testing for principal component analysis or partial least squares. Step 1. PCA: I feel that I have a not-terrible understanding of PCA: You find ...
5
votes
1answer
388 views

What's the best way to choose data for Crossvalidation on linear regression settings (PCA, PLS)

We are extracting features from EEG, which is a time dependent signal. We have signals of 10,000 datapoints over 64 channels, and we extract 10 features per timestamp per channel, so at the end we ...
4
votes
3answers
377 views

What does it mean to say that a regression method is (not) “scale invariant”?

I was just studying partial least squares regression, and I read that it is "not scale invariant". What does "scale invariant" mean, and why is partial least squares, and why would ...
4
votes
1answer
247 views

Under what conditions can a PLS regression model be expressed by single linear equation?

I am confused by two, yet inconsistent for me, facts: Since the PLS regression is expressed by matrices of scores and loadings as $$X=TP^T+E\\Y=UQ^T+F$$ how it can be translated into linear equation ...
4
votes
1answer
3k views

What are guidelines for SmartPLS boostrapping case size?

In SmartPLS, bootstrapping is used to generate the t statistic from which statistical significance can be judged. The two main bootstrapping parameters are case and sample size. Increasing the sample ...
4
votes
1answer
2k views

Combining principal component analysis and partial least squares

I know PCA and PLS are considered as alternative method to each other. But I am thinking about a kind of combination of the two in case of lots of predictors with little variability. In that case, ...
4
votes
1answer
39 views

Multiple dim responding variable the relation between CCA and trivial linear regression (PLSR, CCA, PCA, PCR and Linear Regression)

Here is my summary of Multivariate Linear Regression between explain variable $\textbf{x}$ and responding variable $\textbf{y}.$ I have summaried the relation ...
4
votes
1answer
4k views

Understanding NIPALS algorithm for PLS

I've found a nice presentation describing PLS1 and PLS2 algorithms (pages 16-19). It's pretty clear but there is a thing confusing me. For PLS1. Let's look at the algorithm. The first steps are $w = ...
4
votes
1answer
695 views

Problems with implementing cross-validation for OLS, PLSR and PCR

I am new to regression methods. I am creating Multiple Linear Regression, Partial Least Squares Regression and Principal Component Regression models for my dataset, and I am a bit confused with the ...
4
votes
1answer
971 views

Is it appropriate to perform feature selection before running Partial Least Square (PLS) regression?

I'm doing a series of PLS analysis to test the contribution of a set of environmental variables to the invertebrate community of a river. I am introducing into the model 23 environmental variables (2 ...
4
votes
1answer
2k views

Is the PLS-DA approach for categorical variables the same as that used for PLS regression?

I understand the approach used for partial least squares for regression (PLS regression) where the PLS components are chosen such that the correlation between the scores of the PLS components of the ...
4
votes
2answers
2k views

What's the difference between logistic regression and PLS-DA?

I heard just recently about PLS-DA and I was wondering how it differs from multinomial logistic regression, since logistic regression can be also used for categorical dependent variables.
4
votes
0answers
100 views

Is it possible to combine bayesian SEM with PLS SEM?

I have already read some books about both two structural equation models. It seems both SEMs are suitable to the situation with small observations and large variables. I assume to use combine both ...
3
votes
1answer
1k views

When there is only one dependent variable, is partial least squares regression the same as principal component regression?

When there is only one response (dependent) variable, what is the advantage of partial least squares (PLS) regression over principal component regression (PCR)? My understanding is that PLS is only ...
3
votes
1answer
2k views

How many components to use when doing a PLS regression?

I'm doing a PLS regression with SAS. My man-a asked me to do find the numbers of principal components for the dataset I'm working with through SAS. As I've never done that before, I'm confused on how ...
3
votes
1answer
4k views

Prediction of independent data with PLS

In Matlab's plsregress function and in many other statistic toolboxes, there is a BETA vector returned that simplyfies the regression problem to(excluding the ...
3
votes
2answers
6k views

What is the difference between PCA and PLS-DA?

I read a paper Obesity changes the human gut mycobiome (2015) in which the authors used PLS-DA to look at the differences in their groups based on the microbiome of their gut. I am currently working ...
3
votes
2answers
251 views

What variable importance criterion?

A student of mine developed a heuristic supervised machine learning algorithm for highly multivariate data. It seems to work pretty well, and once the model has been derived from the training data set,...
3
votes
1answer
1k views

Magnitude and direction of relationship between predictors and dependent in regression

I'm doing partial least squares regression (PLSR), using the df below, to investigate how to predictors (catchment characteristics) influence the dependent (nitrogen in the river). In this data.frame, ...
3
votes
1answer
300 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
3
votes
1answer
387 views

RMSE behaviour during cross-validation of a PLS model - What does it mean?

In PLS regression, one of the approaches in selecting the number of components (Latent Variables, LV) is to perform cross-validation over a range of LV, and select the one with lower Root Mean Squared ...
3
votes
1answer
72 views

Formulating Partial Least Squares as minimizing squared error

The book chapter linked below (see section 4.3.1) lists a few formulations of partial least squares (PLS). The first two make sense to me and seem standard: $$\underset{\mathbf{u}, \mathbf{v}}{\text{...
3
votes
1answer
470 views

Does it make sense to run a stepwise regression on components estimated through Partial Least Square?

I am trying to solve a problem of dimentionality reduction on a Matrix of predictors X(136x481). I found that PCA does not a good job in my case because it create components that explain just the ...
3
votes
1answer
2k views

Choosing the number of PLSR components

I am trying to choose how many components to retain in my PLSR. My total variance explained for the response variable is only about 30%, and the first 2 components explain 99% of this. Intuitively I ...
3
votes
2answers
540 views

PLS with more variables than data points

Does it make sense to run partial least squares (PLS) on a data set that has many more columns (variables) than it has rows (data points)? I am using plsr in R. I ...
3
votes
1answer
208 views

What is the origin of the PLS1 algorithm given on the PLS Wikipedia page?

The Wikipedia page for Partial Least Squares (PLS) gives an algorithm for the method which is uncited and for which I cannot find the source material. It appears to be very much simpler than most if ...
3
votes
1answer
169 views

Screening data prior to PCA v. PLS

I have a very large time series matrix $X$, where the number of observations (rows) $n$ is much smaller than the number of input variables (columns) $p$. My aim is to use the information in $X$ to ...

1
2 3 4 5