I know the PCA theory and how keeping the k-first principal components help dimensionality reduction. But what about keeping the base vectors ?

You start with $n$ row vectors $x_1,...,x_n$, standardize them, then find the covariance matrix $C$ and diagonalise it $C=VDV^{-1}$, with $D_{ij}=\lambda_j\delta_{ij}$. Then after ordering the eigenvector with decreasing eigenvalue order you can use the first $p$ eigenvectors as a base to express your starting variables. The process will ensure you explain a maximum of the variance while only keeping $p<n$ variables. The main downside I see to the process is the somewhat lack of transparency of the eigenvectors, starting variables have values, units, meanings, that are usefull to deal with on a daily basis.

Is there any way to express the proportion of the variance explained by one of the starting vector ? by a subset of $p$ of the starting vectors ?

Is there any process to select $p$ of the starting vectors to maximise the variance explained by this subset ? Would this kind of process be incremental ? (ie. can the selection of $p+1$ vectors that maximise the explained variance can be built by adding one vector to the selection of $p$ vectors that maximised the explained variance ?).

  • 1
    $\begingroup$ You are very nearly describing stepwise regression, so that's a good term to research for answers. $\endgroup$ – whuber 2 days ago
  • $\begingroup$ Ok thanks, i'll look into that. What about my first question ? Is it possible to express the proportion of variance explained by a subset of variables ? $\endgroup$ – Lcrmorin 2 days ago
  • $\begingroup$ That's exactly what $R^2$ does. $\endgroup$ – whuber 2 days ago

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.