Linked Questions
25
votes
3
answers
17k
views
Relationship between ridge regression and PCA regression
I remember having read somewhere on the web a connection between ridge regression (with $\ell_2$ regularization) and PCA regression: while using $\ell_2$-regularized regression with hyperparameter $\...
- 478
14
votes
3
answers
5k
views
Why do we use PCA to speed up learning algorithms when we could just reduce the number of features?
In a machine learning course, I learned that one common use of PCA (Principal Component Analysis) is to speed up other machine learning algorithms. For example, imagine you are training a logistic ...
- 406
23
votes
2
answers
6k
views
Low variance components in PCA, are they really just noise? Is there any way to test for it?
I'm trying to decide if a component of a PCA shall be retained, or not. There are a gazillion of criteria based on the magnitude of the eigenvalue, described and compared e.g. here or here.
However, ...
- 231
13
votes
3
answers
5k
views
The first principal component does not separate classes, but other PCs do; how is that possible?
I ran PCA on 17 quantitative variables in order to obtain a smaller set of variables, that is principal components, to be used in supervised machine learning for classifying instances into two classes....
- 155
13
votes
2
answers
8k
views
How to use principal components analysis to select variables for regression?
I am currently using principal components analysis to select variables to use in modelling. At the moment, I make measurements A, B and C in my experiments -- What I really want to know is: Can I ...
- 1,975
9
votes
1
answer
13k
views
Understanding cluster plot and component variability
I have run k-means clustering. I have also plotted the results using the following code in R:
...
- 389
12
votes
2
answers
4k
views
What is the advantage of reducing dimensionality of predictors for the purposes of regression?
What are the applications or advantages of dimension reduction regression (DRR) or supervised dimensionality reduction (SDR) techniques over traditional regression techniques (without any ...
- 2,545
13
votes
1
answer
2k
views
Why does Daniel Wilks (2011) say that principal component regression "will be biased"?
In Statistical Methods in the Atmospheric Sciences, Daniel Wilks notes that multiple linear regression can lead to problems if there are very strong intercorrelations among the predictors (3rd edition,...
- 1,439
3
votes
1
answer
6k
views
What is the difference between PCA + Linear Regression versus PCR?
I am trying to do linear regression to predict the time a user spends listening to music using the following dataset:
My end goal is to know which characteristics or columns lead to higher listening. ...
- 219
5
votes
1
answer
7k
views
How to use principal components as predictors in regression?
I have a couple of questions involving doing a regression (logistic or linear) after principal component analysis.
If I find principal components using Principal component analysis, can I use these ...
- 193
6
votes
2
answers
2k
views
Is dimensionality reduction almost always useful for classification?
Is singular value decomposition almost always useful in practice for enhancing the predicative power of a trained classification model?
E.x. A dataset for classification has 20,000 features. Run SVD ...
- 848
8
votes
1
answer
587
views
Why is my LDA performance a non-monotonic function of the amount of training data?
Short story:
I have a classification pipeline consisting of some feature extractors and an LDA classifier. When evaluating the pipeline in a cross-validation I get a decent test accuracy of 94% (for ...
- 133
3
votes
0
answers
466
views
Random Forest after PCA results don't make sense [duplicate]
I'm playing around with random forest classification and principal component analysis using scikit-learn and have found a point of confusion. I want to fit two models that predict 1 of 5 target ...
- 131
2
votes
0
answers
421
views
Do the problems of stepwise variable selection exist in FA, PCA, SEM?
Note: This is a revision of my original question.
I have read the critique of stepwise variable selection and "all possible subsets regression" by Professor Frank Harrell here.
Are factor analysis, ...
- 96
1
vote
1
answer
160
views
Do we have different results by applying Principal component Regression for different dependent?
I have several dependent variables to analysis. Since the independent variables are highly correlated, I am thinking about using the PCR model. Now I was wondering if by choosing different dependent ...
- 87