From a very general point of view, when you have a dataset $X$ and want to predict a label $y$, what is the purpose of beginning with a PCA (principal component analysis) first, and then doing the prediction itself (with logistic regression, or random forest or whatever) from both intuitive and theoretical reason? In which case can this improve the quality of prediction?
$\begingroup$
$\endgroup$
0
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
If you have variables that correlate between themselves, they provide only a little meaningful information by themselves, but you can clump them together. It decreases the amount of explaining variables.
Benefits:
- Training a predictor is faster. With big data sizes this matters.
- You can visualize higher dimensional data, by creating new variables that explain more. For example, you can find something called beauty of a day from temperature, sold ice cream and drowning deaths (classical set of parameters used to demonstrate unrelated correlations).
$\begingroup$
$\endgroup$
PCA can be used as a step for pre-processing the model features. From my experience: Pros:
- Reducing dimensionality.
- Reducing noise and possibly improving model performance.
Cons:
- Sensitivity to original features scale.
- Transformed space new features might not have any representation in the real world, and reverse transformation might be messy.
$\begingroup$
$\endgroup$
PRobably already answered here: The biggest reason is to reduce dimensionality - computing logistic regression on 100 features can be cumbersome. PCA will recreate this feature matrix in terms of new variables that capture direction of variance in the data. So You are working with variables that cause the data to move the most. And in general first few components can capture very large proportion of the overall variance, say 90+ %. Then you can only work with those few aritificial new variables instead of original ones. You only need to transform everything to those new variables - nice exercise in linear algebra :)
