I know the PCA theory and how keeping the k-first principal components help dimensionality reduction. But what about keeping the base vectors ?

You start with $n$ row vectors $x_1,...,x_n$, standardize them, then find the covariance matrix $C$ and diagonalise it $C=VDV^{-1}$, with $D_{ij}=\lambda_j\delta_{ij}$. Then after ordering the eigenvector with decreasing eigenvalue order you can use the first $p$ eigenvectors as a base to express your starting variables. The process will ensure you explain a maximum of the variance while only keeping $p<n$ variables. The main downside I see to the process is the somewhat lack of transparency of the eigenvectors, starting variables have values, units, meanings, that are usefull to deal with on a daily basis.

Is there any way to express the proportion of the variance explained by one of the starting vector ? by a subset of $p$ of the starting vectors ?

Is there any process to select $p$ of the starting vectors to maximise the variance explained by this subset ? Would this kind of process be incremental ? (ie. can the selection of $p+1$ vectors that maximise the explained variance can be built by adding one vector to the selection of $p$ vectors that maximised the explained variance ?).

  • 1
    $\begingroup$ You are very nearly describing stepwise regression, so that's a good term to research for answers. $\endgroup$ – whuber Feb 13 at 22:38
  • $\begingroup$ Ok thanks, i'll look into that. What about my first question ? Is it possible to express the proportion of variance explained by a subset of variables ? $\endgroup$ – were_cat Feb 14 at 10:05
  • $\begingroup$ That's exactly what $R^2$ does. $\endgroup$ – whuber Feb 14 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.