Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Whats the difference between 2 features and 2 principal components? I know what a PCA is, I just have the following problem:
If my data has 2 features, the PCA will produce 2 components. So why does the predictive power of 2 features equal the predicive power of 2 principal components?
For linear models, representing your features in your original two-dimensional basis or a new two-dimensional basis from PCA won't change the predictive power. For the purposes of this question, there's nothing special about the PCA basis.
Let $\mathbf{x} = \left[ \begin{array}{c} x_1 \\ x_2 \end{array} \right]$ denote an observation from a two-dimensional feature space.
We can construct a new basis for this two-dimensional space using any set of two linearly independent vectors. These vectors could be from Principal Component Analysis (PCA), but they could also come from elsewhere (as long as they're linearly independent). Let $U$ be a matrix whose columns are vectors of the new basis. Coordinates of our observation $\mathbf{x}$ in terms of the new basis is given by multiplying by the inverse of the matrix $U$.
$$\mathbf{z} = U^{-1} \mathbf{x}$$
In many machine learning contexts, you use linear transformations of your observations to do stuff. You use some $A\mathbf{x}$ where matrix $A$ represents some linear map.
Clearly:
$$ A \mathbf{x} = AUU^{-1}\mathbf{x} = AU \mathbf{z}$$
That is, it's entirely equivalent to use:
You can see though that this wouldn't hold for non-linear transformations.
Model is: $$y_i = \mathbf{x}_i \cdot \boldsymbol{\beta} + \epsilon_i$$
OLS estimate: $$ \mathbf{b}_x = (X'X)^{-1}(X'\mathbf{y})$$ With data in new basis: $Z = X{U'}^{-1}$
$$ \begin{align*} \mathbf{b}_z &= (Z'Z)^{-1}(Z'\mathbf{y}) \\ &= \left( U^{-1}X'X {U'}^{-1} \right) ^{-1} {U^{-1}X' \mathbf{y} }\\ &= U' \left( X'X \right) ^{-1} U U^{-1}X' \mathbf{y} \\ &= U' \left( X'X \right) ^{-1} X' \mathbf{y} \\ &= U' \mathbf{b}_x \\ \end{align*} $$
Keep going and the residuals will be the same etc... You have the same predictive power and the estimated coefficients are related by the change in basis.
In your question, I believe features refers to the size of the space. For example, measurements for population variation were taken on two features: hair color and shoe size.
Principle component analysis is often can often be associated with dimension reduction, but it need not be. In it's most general formulation, it is simply a rotation of the space to the most coordinate system where the variances along the axes are uncorrelated. In such a coordinate system, the components are linear combinations of the features, and in some cases, they may be the features themselves.
The questions is asking you why using all of the features and using all of the principle components provides the same predictive power. I'll let you work that part out yourself.
Without any context, if you have two noncollinear variables, you have two features and two PCs. The PC reflects the structure of the data. Multiple collinear variables lead to fewer PCs than variables, the PCs simply provide an orthonormal basis for a matrix of features.
What is perhaps interesting is sparse representation. LASSO is a supervised learning approach where a small number of features can be selected for predictive value. Alternately, choosing a small number of PCs and their loadings provides sparse representation of a matrix of features in an unsupervised approach. In that sense, the implications would be vastly different.
