Questions tagged [partial-least-squares]

A class of linear methods for modeling the relationship between two groups of variables, X and Y. Includes PLS regression.

Filter by
Sorted by
Tagged with
33
votes
2answers
5k views

Theory behind partial least squares regression

Can anyone recommend a good exposition of the theory behind partial least squares regression (available online) for someone who understands SVD and PCA? I have looked at many sources online and have ...
28
votes
1answer
7k views

PCA, LDA, CCA, and PLS

How are PCA, LDA, CCA, and PLS related? They all seem "spectral" and linear algebraic and very well understood (say 50+ years of theory built around them). They are used for very different things (PCA ...
21
votes
2answers
980 views

The limit of “unit-variance” ridge regression estimator when $\lambda\to\infty$

Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
16
votes
2answers
2k views

What is the “partial” in partial least squares methods?

In partial least squares regression (PLSR) or partial least squares structural equation modelling (PLS-SEM), what does the term "partial" refer to?
16
votes
1answer
3k views

What is the connection between partial least squares, reduced rank regression, and principal component regression?

Are reduced rank regression and principal component regression just special cases of partial least squares? This tutorial (Page 6, "Comparison of Objectives") states that when we do partial least ...
15
votes
1answer
8k views

Regression in $p>n$ setting: how to choose regularization method (Lasso, PLS, PCR, ridge)?

I am trying see whether to go for ridge regression, LASSO, principal component regression (PCR), or Partial Least Squares (PLS) in a situation where there are large number of variables / features ($p$)...
13
votes
2answers
5k views

Model assumptions of partial least squares (PLS) regression

I am trying to find information regarding the assumptions of PLS regression (single $y$). I am especially interested in a comparison of the assumptions of PLS with regards to those of OLS regression. ...
12
votes
1answer
21k views

Partial least squares regression in R: why is PLS on standardized data not equivalent to maximizing correlation?

I am very new in partial least squares (PLS) and I try to understand the output of the R function plsr() in the pls package. Let ...
11
votes
1answer
21k views

What is the difference between “loadings” and “correlation loadings” in PCA and PLS?

One common thing to do when doing Principal Component Analysis (PCA) is to plot two loadings against each other to investigate the relationships between the variables. In the paper accompanying the ...
10
votes
1answer
3k views

Why do all the PLS components together explain only a part of the variance of the original data?

I have a dataset consisting of 10 variables. I ran partial least squares (PLS) to predict a single response variable by these 10 variables, extracted 10 PLS components, and then computed the variance ...
10
votes
2answers
4k views

How to compute the confidence intervals on regression coefficients in PLS?

The underlying model of PLS is that a given $n \times m$ matrix $X$ and $n$ vector $y$ are related by $$X = T P' + E,$$ $$y = T q' + f,$$ where $T$ is a latent $n \times k$ matrix, and $E, ...
10
votes
1answer
1k views

Difference between PLS regression and PLS path modeling. Criticism of PLS

This question was asked here but no one gave a good answer. So I think it's a good idea to bring it up again and also I would like to add some more comments/questions. The first question is what is ...
7
votes
1answer
358 views

Probabilistic models for partial least squares, reduced rank regression, and canonical correlation analysis?

This question results from the discussion following a previous question: What is the connection between partial least squares, reduced rank regression, and principal component regression? For ...
6
votes
1answer
9k views

How to use PCA in regression?

I'm currently reading in the Applied Predictive Modeling book about PCA for dimensionality reduction. I've read the following: If the predictive relationship between the predictors and response is ...
6
votes
1answer
3k views

Is each of the PCA or PLS components just one of the original variables?

I am confused about what a component is in PCA and PLS. Are the components just the original variables but not necessarily in the same order? For example, in PCA, if I had 8 variables in my data, ...
6
votes
1answer
1k views

Mean centering or not in the context of Partial Least Squares

In my current project, I'm using PLS regression on infrared spectra (FTIR). For this I'm using R and the pls function from the plsr package. ...
6
votes
1answer
641 views

What does “orthogonalize” mean?

I have read in The Elements of Statistical Learning book and particularly in the Partial Least Squares (PLS) section: Orthogonalize each $x_j^{(m−1)}$ with respect to $z_m$. I would like to know ...
6
votes
1answer
1k views

Measuring predictive accuracy for multiple dependent variables

In machine learning and in statistics there exist plenty of measures which estimate the performance of a predictive model. For example, classification accuracy, area under ROC curve ... for ...
6
votes
1answer
5k views

Validation metrics (R2 and Q2) for Partial Least Squares (PLS) Regression

I'm attempting to validate my Partial Least Squares (PLS) -regression model. From documentation and other readings regarding PLS regression I've come to understand that there are generally two ...
6
votes
1answer
5k views

PLS (partial least squares) weights, loadings, and scores interpretations

In SKLearn PLSRegression, several items can be called after a model is trained: Loadings Scores Weights All the above are separated by X and Y I intuitively understand that x_scores and y_scores ...
6
votes
1answer
2k views

How do you predict the value of new instance, when the training data were normalized?

I estimated a Partial Least Squares model where the X matrix had normalized columns. Now I want to predict the value for a new instance (which is a frequency vector summing to one.) I assume that if I ...
6
votes
0answers
390 views

What, if any, dissimilarity is preserved in partial least squares (PLS)?

When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
5
votes
1answer
380 views

How would Y-aware PCA for binaries look?

I recently stumbled upon Y-aware PCA in the blog of win-vector. They describe how PCA can be adjusted not to explain variation in $X$ but covariation of $X$ and $Y$. This is explained for the case ...
5
votes
1answer
8k views

PCA and PLS: testing variables for significance

I'm trying to understand the process for statistical testing for principal component analysis or partial least squares. Step 1. PCA: I feel that I have a not-terrible understanding of PCA: You find ...
5
votes
1answer
372 views

What's the best way to choose data for Crossvalidation on linear regression settings (PCA, PLS)

We are extracting features from EEG, which is a time dependent signal. We have signals of 10,000 datapoints over 64 channels, and we extract 10 features per timestamp per channel, so at the end we ...
4
votes
1answer
233 views

Under what conditions can a PLS regression model be expressed by single linear equation?

I am confused by two, yet inconsistent for me, facts: Since the PLS regression is expressed by matrices of scores and loadings as $$X=TP^T+E\\Y=UQ^T+F$$ how it can be translated into linear equation ...
4
votes
1answer
3k views

What are guidelines for SmartPLS boostrapping case size?

In SmartPLS, bootstrapping is used to generate the t statistic from which statistical significance can be judged. The two main bootstrapping parameters are case and sample size. Increasing the sample ...
4
votes
1answer
2k views

Combining principal component analysis and partial least squares

I know PCA and PLS are considered as alternative method to each other. But I am thinking about a kind of combination of the two in case of lots of predictors with little variability. In that case, ...
4
votes
1answer
609 views

Problems with implementing cross-validation for OLS, PLSR and PCR

I am new to regression methods. I am creating Multiple Linear Regression, Partial Least Squares Regression and Principal Component Regression models for my dataset, and I am a bit confused with the ...
4
votes
1answer
3k views

Understanding NIPALS algorithm for PLS

I've found a nice presentation describing PLS1 and PLS2 algorithms (pages 16-19). It's pretty clear but there is a thing confusing me. For PLS1. Let's look at the algorithm. The first steps are $w = ...
4
votes
1answer
616 views

Is it appropriate to perform feature selection before running Partial Least Square (PLS) regression?

I'm doing a series of PLS analysis to test the contribution of a set of environmental variables to the invertebrate community of a river. I am introducing into the model 23 environmental variables (2 ...
4
votes
0answers
93 views

Is it possible to combine bayesian SEM with PLS SEM?

I have already read some books about both two structural equation models. It seems both SEMs are suitable to the situation with small observations and large variables. I assume to use combine both ...
3
votes
1answer
1k views

When there is only one dependent variable, is partial least squares regression the same as principal component regression?

When there is only one response (dependent) variable, what is the advantage of partial least squares (PLS) regression over principal component regression (PCR)? My understanding is that PLS is only ...
3
votes
1answer
2k views

How many components to use when doing a PLS regression?

I'm doing a PLS regression with SAS. My man-a asked me to do find the numbers of principal components for the dataset I'm working with through SAS. As I've never done that before, I'm confused on how ...
3
votes
2answers
4k views

What is the difference between PCA and PLS-DA?

I read a paper Obesity changes the human gut mycobiome (2015) in which the authors used PLS-DA to look at the differences in their groups based on the microbiome of their gut. I am currently working ...
3
votes
2answers
239 views

What variable importance criterion?

A student of mine developed a heuristic supervised machine learning algorithm for highly multivariate data. It seems to work pretty well, and once the model has been derived from the training data set,...
3
votes
1answer
824 views

Magnitude and direction of relationship between predictors and dependent in regression

I'm doing partial least squares regression (PLSR), using the df below, to investigate how to predictors (catchment characteristics) influence the dependent (nitrogen in the river). In this data.frame, ...
3
votes
1answer
137 views

RMSE behaviour during cross-validation of a PLS model - What does it mean?

In PLS regression, one of the approaches in selecting the number of components (Latent Variables, LV) is to perform cross-validation over a range of LV, and select the one with lower Root Mean Squared ...
3
votes
1answer
61 views

Formulating Partial Least Squares as minimizing squared error

The book chapter linked below (see section 4.3.1) lists a few formulations of partial least squares (PLS). The first two make sense to me and seem standard: $$\underset{\mathbf{u}, \mathbf{v}}{\text{...
3
votes
1answer
394 views

Does it make sense to run a stepwise regression on components estimated through Partial Least Square?

I am trying to solve a problem of dimentionality reduction on a Matrix of predictors X(136x481). I found that PCA does not a good job in my case because it create components that explain just the ...
3
votes
1answer
1k views

Choosing the number of PLSR components

I am trying to choose how many components to retain in my PLSR. My total variance explained for the response variable is only about 30%, and the first 2 components explain 99% of this. Intuitively I ...
3
votes
1answer
276 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
3
votes
1answer
1k views

Is the PLS-DA approach for categorical variables the same as that used for PLS regression?

I understand the approach used for partial least squares for regression (PLS regression) where the PLS components are chosen such that the correlation between the scores of the PLS components of the ...
3
votes
1answer
934 views

Calculate the variance explained in matrix Y by matrix X

I have two matrices corresponding to the same set of $n$ samples, with $j$ and $k$ variables, respectively ($j > 10000$, $k > 10000$). $X$ is an $n \times j$ matrix and $Y$ is an $n \times k$ ...
3
votes
0answers
110 views

Partial Least Squares Regression : deflation of the Y matrix

I am digging deep into the PLSR algorithms and while I have found multiple flavours of if (different normalisations, SIMPLS,..), there is always something in the Y deflation that is throwing me off ...
3
votes
0answers
196 views

PLS “Path Modeling” vs PLS “Regression” [duplicate]

There's a Wiki page for PLS "Path Modeling" and another one for PLS "Regression"... they seem to be saying the same thing, except the first link says PLS is great, and the second says that PLS is ...
3
votes
0answers
413 views

PLS using a kernel matrix

I would like to use a kernel matrix generated with a custom kernel function to fit a PLS-DA model (I am thinking of caret's PLS-DA at the moment), with only one binary response variable in the Y block....
3
votes
0answers
2k views

What is partial least squares (PLS) regression and how is it different from OLS? [duplicate]

Assume we have a simple linear regression model expressed as $$y= X \beta + \epsilon,$$ where $y$ is a vector of size $n \times 1$, $X$ is a matrix of size $ n \times p$, $\beta$ is the regression ...
3
votes
0answers
453 views

How to choose between different options in partial least square regression?

There seem to be several methods of performing partial least square regression. For example in pls pacakge in R, following are available: ...
3
votes
0answers
899 views

Multivariate outlier detection for PLS model

I am working with a PLS model (library pls) in R, where I am developing calibration models for NIRS data. I have been using other commercial software before that allowed me to detect outliers based on ...

1 2 3 4 5