3
$\begingroup$

Principal components analysis finds the axis(/es) along which the variance of some dataset is maximized. What if I'm in a situation where I have a single dependent variable $v$ that is dependent on $n$ independent variables $v = f(x_1, x_2, \dots, x_n)$ and I would like to know "what principal components of $\mathbf x$ cause the most variance in $v$?" Is this an answered question already? I feel like it would be just a small modification of the PCA algorithm..

$\endgroup$
3
  • 1
    $\begingroup$ The answer is that none of the PCs of $X$ may be associated with $v.$ Ordinary least squares regression can be interpreted as finding a linear combination of columns of $x$ that maximize the correlation with $v.$ $\endgroup$
    – whuber
    Sep 1, 2019 at 21:43
  • 1
    $\begingroup$ @whuber is it exactly the same problem as iterated OLS then? $\endgroup$
    – Mike Flynn
    Sep 1, 2019 at 21:56
  • 1
    $\begingroup$ @whuber also, to clarify a possible misunderstanding - by "principal components of $\mathbf x$" I do not mean the p.c.s as identified by PCA, rather I mean "set of axes that give directions that drive the most variance of $v$". This is probably related to the output of ordinary least squares regression. $\endgroup$
    – Mike Flynn
    Sep 1, 2019 at 22:05

1 Answer 1

1
$\begingroup$

If I got your question correctly (and I am not sure), the solution is that you have to take the spectral decomposition $VDV^{T}$ where V is the orthonormal matrix of eigenvectors and D is the diagonal matrix of eigenvalues. Then you transform the set of dependent variables by multiplying the original matrix of predictors $X^{*}$ by the eigenvectors $X=X^{*}V$. Then you can regress y against the new transformer predictors X that are now uncorrelated. Since they are uncorrelated, the total explained variance of y will be the sum of variances of $\beta x$ where each x is the vector representing each column of X (so each transformed variable) and $\beta$ denotes its corresponding beta estimated at previous step. The PC variable $x^{max}$ such that $\beta^{2}_{x^{max}} Var(x^{max})$ is maximum among all the pc transformed variables is the one explaining the highest portion of the variance of $y$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.