Linked Questions
16
votes
4
answers
25k
views
How does PCA improve the accuracy of a predictive model? [duplicate]
I've seen in a kaggle challenge about digit recognition someone who used PCA before decision tree or other techniques.
I thought it was just for compressing data but he aimed to improve his score.
...
- 193
15
votes
2
answers
27k
views
PCA before Random Forest Regression provide better predictive scores for my dataset than just Random Forest Regression, how to explain it? [duplicate]
I have a regression dataset where the features are on the order of ~ 400 variables and the dataset itself is around 300 samples. I tried to use Random Forest Regression (RFR) on the dataset and used ...
- 165
6
votes
1
answer
473
views
How can preprocessing with PCA but keeping the same dimensionality improve random forest results? [duplicate]
I found these sentences:
PCA before random forest can be useful not for dimensionality
reduction but to give you data a shape where random forest can perform
better.
I am quiet sure that ...
1
vote
3
answers
280
views
Why and when would one begin with PCA on $X$ when predicting $y$? [duplicate]
From a very general point of view, when you have a dataset $X$ and want to predict a label $y$, what is the purpose of beginning with a PCA (principal component analysis) first, and then doing the ...
- 11
1
vote
0
answers
404
views
Is it realistic to achieve better results when doing PCA before neural network classification? [duplicate]
I'm doing a data analysis on data with more than 100 dimensions.
After that different ML-Algorithms like NN are applied to it.
When I do a PCA in the first place to reduce dimensionality to somewhat ...
- 71
1
vote
0
answers
309
views
When using principal components as predictors in linear regression, PC1 comes out not significant [duplicate]
I have a dataset with 37 independent variables and a dependent variable. In order to take care of multicollinearity among the independent variables, I conducted a PCA on them. My first principal ...
1
vote
0
answers
92
views
How to efficiently do PCA/SVD on dataset with thousands of features (both continuous and OHE) [duplicate]
I am currently dealing with a dataset with about 300,000 records, there are a wide variety of categories in several columns and naturally when one-hot-encoding these the number of features increases ...
- 11
1
vote
0
answers
73
views
Adding more features improves the variance explained by PCA but the prediction model performs worse [duplicate]
With my 37 base features, I obtain a PCA 7 components whose explained variance is 62584.661. Below is my code (I use scikit-learn):
...
- 877
0
votes
0
answers
16
views
PCA on predictor variables before supervised learning is appplied [duplicate]
Say I have 20 predictor vars (X's) and 1 response var (Y) and I'm attempting to build a supervised model y=f(x). Is it advisable or is it "OK" to firstly run PCA on all of the Predictor variables - ...
- 835
33
votes
5
answers
23k
views
What can cause PCA to worsen results of a classifier?
I have a classifier that I'm doing cross-validation on, along with a hundred or so features that I'm doing forward selection on to find optimal combinations of features. I also compare this against ...
- 779
43
votes
6
answers
14k
views
Examples of PCA where PCs with low variance are "useful"
Normally in principal component analysis (PCA) the first few PCs are used and the low variance PCs are dropped, as they do not explain much of the variation in the data.
However, are there examples ...
- 433
64
votes
3
answers
13k
views
Why does shrinkage work?
In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive ...
33
votes
4
answers
14k
views
Is PCA always recommended?
I was wondering if PCA can be always applied for dimensionality reduction before a classification or regression problem. My intuition tells me that the answer is no.
If we perform PCA then we ...
- 744
28
votes
6
answers
17k
views
Dealing with correlated regressors
In a multiple linear regression with highly correlated regressors, what is the best strategy to use? Is it a legitimate approach to add the product of all the correlated regressors?
- 1,579
33
votes
2
answers
28k
views
Does it make sense to combine PCA and LDA?
Assume I have a dataset for a supervised statistical classification task, e.g., via a Bayes' classifier. This dataset consists of 20 features and I want to boil it down to 2 features via ...
user39663
25
votes
3
answers
17k
views
Relationship between ridge regression and PCA regression
I remember having read somewhere on the web a connection between ridge regression (with $\ell_2$ regularization) and PCA regression: while using $\ell_2$-regularized regression with hyperparameter $\...
- 478
14
votes
3
answers
5k
views
Why do we use PCA to speed up learning algorithms when we could just reduce the number of features?
In a machine learning course, I learned that one common use of PCA (Principal Component Analysis) is to speed up other machine learning algorithms. For example, imagine you are training a logistic ...
- 406
23
votes
2
answers
6k
views
Low variance components in PCA, are they really just noise? Is there any way to test for it?
I'm trying to decide if a component of a PCA shall be retained, or not. There are a gazillion of criteria based on the magnitude of the eigenvalue, described and compared e.g. here or here.
However, ...
- 231
13
votes
3
answers
5k
views
The first principal component does not separate classes, but other PCs do; how is that possible?
I ran PCA on 17 quantitative variables in order to obtain a smaller set of variables, that is principal components, to be used in supervised machine learning for classifying instances into two classes....
- 155
13
votes
2
answers
8k
views
How to use principal components analysis to select variables for regression?
I am currently using principal components analysis to select variables to use in modelling. At the moment, I make measurements A, B and C in my experiments -- What I really want to know is: Can I ...
- 1,975
9
votes
1
answer
13k
views
Understanding cluster plot and component variability
I have run k-means clustering. I have also plotted the results using the following code in R:
...
- 389
12
votes
2
answers
4k
views
What is the advantage of reducing dimensionality of predictors for the purposes of regression?
What are the applications or advantages of dimension reduction regression (DRR) or supervised dimensionality reduction (SDR) techniques over traditional regression techniques (without any ...
- 2,545
13
votes
1
answer
2k
views
Why does Daniel Wilks (2011) say that principal component regression "will be biased"?
In Statistical Methods in the Atmospheric Sciences, Daniel Wilks notes that multiple linear regression can lead to problems if there are very strong intercorrelations among the predictors (3rd edition,...
- 1,439
3
votes
1
answer
6k
views
What is the difference between PCA + Linear Regression versus PCR?
I am trying to do linear regression to predict the time a user spends listening to music using the following dataset:
My end goal is to know which characteristics or columns lead to higher listening. ...
- 219
5
votes
1
answer
7k
views
How to use principal components as predictors in regression?
I have a couple of questions involving doing a regression (logistic or linear) after principal component analysis.
If I find principal components using Principal component analysis, can I use these ...
- 193
6
votes
2
answers
2k
views
Is dimensionality reduction almost always useful for classification?
Is singular value decomposition almost always useful in practice for enhancing the predicative power of a trained classification model?
E.x. A dataset for classification has 20,000 features. Run SVD ...
- 848
8
votes
1
answer
587
views
Why is my LDA performance a non-monotonic function of the amount of training data?
Short story:
I have a classification pipeline consisting of some feature extractors and an LDA classifier. When evaluating the pipeline in a cross-validation I get a decent test accuracy of 94% (for ...
- 133
3
votes
0
answers
466
views
Random Forest after PCA results don't make sense [duplicate]
I'm playing around with random forest classification and principal component analysis using scikit-learn and have found a point of confusion. I want to fit two models that predict 1 of 5 target ...
- 131
2
votes
0
answers
421
views
Do the problems of stepwise variable selection exist in FA, PCA, SEM?
Note: This is a revision of my original question.
I have read the critique of stepwise variable selection and "all possible subsets regression" by Professor Frank Harrell here.
Are factor analysis, ...
- 96
1
vote
1
answer
160
views
Do we have different results by applying Principal component Regression for different dependent?
I have several dependent variables to analysis. Since the independent variables are highly correlated, I am thinking about using the PCR model. Now I was wondering if by choosing different dependent ...
- 87
0
votes
0
answers
321
views
Why do PCA scores in a regression lead to higher accuracy than raw variables?
I have a dataset that has some 25 continuous variables and a continuous target. A tutor showed using this dataset, that when PCA scores are used instead of raw variables as inputs for predicting ...
- 384
2
votes
0
answers
236
views
How to interpret PCA loading and to relate it with correlation coefficient of associated independent variable with dependent variable
There are 2 sides to this question. On the first side, we are trying to run PCA.
While running PCA on a dataset having 62 independent variables (IVs), I have found that first component (PC1) explains ...
- 131
2
votes
0
answers
191
views
How can a later principal component be significant predictor in a regression, when an earlier PC is not? [duplicate]
I have a question regarding principal component regression (regression of a DV on principal components). I have 4 components in my PCR and the third component is non-significant as a predictor. What ...
- 21
2
votes
0
answers
67
views
Reference for this claim: important features in data can be "hidden" in the higher PCA axes that are typically thrown out [duplicate]
I remember reading a paper a while ago that demonstrated some cases in which PCA would fail to capture important features of a data set in the first few principal components, but where those features ...
- 12.8k
3
votes
0
answers
44
views
Example datasets where PCA could improve or decrease performance of SVM? [duplicate]
Going through the top answers in How can top principal components retain the predictive power on a dependent variable (or even lead to better predictions)?, I understand that doing PCA and keeping the ...
- 1,209