Linked Questions

2
votes
0answers
1k views

What is partial least squares (PLS) regression and how is it different from OLS? [duplicate]

Assume we have a simple linear regression model expressed as $$y= X \beta + \epsilon,$$ where $y$ is a vector of size $n \times 1$, $X$ is a matrix of size $ n \times p$, $\beta$ is the regression ...
2
votes
0answers
45 views

A mathematical description of Partial Least Squares (PLS)? [duplicate]

I have read many articles about PLS, but I could not understand the mathematical description yet. I know that it is quite similar to principal component regression (PCR), except that it takes into ...
55
votes
2answers
8k views

Why does shrinkage work?

In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive ...
21
votes
2answers
866 views

The limit of “unit-variance” ridge regression estimator when $\lambda\to\infty$

Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
9
votes
1answer
7k views

Principal Component Analysis Eliminate Noise In The Data

Do Principal Component Analysis (PCA) eliminate noise in the data set? If PCA do not eliminate noise in the data set, what actually does PCA do to the data set? Can somebody help me regarding this ...
13
votes
2answers
4k views

Model assumptions of partial least squares (PLS) regression

I am trying to find information regarding the assumptions of PLS regression (single $y$). I am especially interested in a comparison of the assumptions of PLS with regards to those of OLS regression. ...
5
votes
1answer
3k views

Is each of the PCA or PLS components just one of the original variables?

I am confused about what a component is in PCA and PLS. Are the components just the original variables but not necessarily in the same order? For example, in PCA, if I had 8 variables in my data, ...
9
votes
1answer
1k views

Difference between PLS regression and PLS path modeling. Criticism of PLS

This question was asked here but no one gave a good answer. So I think it's a good idea to bring it up again and also I would like to add some more comments/questions. The first question is what is ...
3
votes
1answer
1k views

Objective function of canonical correlation analysis (CCA)

Given two vectors of random variables $X$ and $Y$, Canonical Correlation Analysis (CCA) finds the transformation matrices $A$ and $B$ so that $\operatorname{corr}(A_{1*} X, B_{1*} Y)$ is first maximal,...
4
votes
1answer
350 views

How would Y-aware PCA for binaries look?

I recently stumbled upon Y-aware PCA in the blog of win-vector. They describe how PCA can be adjusted not to explain variation in $X$ but covariation of $X$ and $Y$. This is explained for the case ...
4
votes
1answer
561 views

Problems with implementing cross-validation for OLS, PLSR and PCR

I am new to regression methods. I am creating Multiple Linear Regression, Partial Least Squares Regression and Principal Component Regression models for my dataset, and I am a bit confused with the ...
1
vote
1answer
291 views

Combining PLS-DA with PCA dimension reduction

I am implementing the PLS-DA method presented here on a data set and I am trying to understand the procedure; and whether there is anything conceptually wrong in my steps. I start with $188 \times ...
1
vote
0answers
139 views

Does PLS have a corresponding objective function to PCA's?

Paraphrased from Understanding Machine Learning by Shalev-Shwartz: Let $\mathbf{x}_1, \dots, \mathbf{x}_m \in \mathbb{R}^d$, $\mathbf{W}$ an $n \times d$ matrix with real entries, and $\mathbf{U}$...