The answer might be given in even simpler terms: the multiple regression has one step more than the pca if seen in terms of linear algebra, and from the second step the instability comes into ...

For understanding this I always prefer the cholesky-decomposition of the correlation-matrix. Assume the correlation-matrix R of the three variable $X.Y.Z$ as $$\text{ R =} \left[ \begin{array} {rrr} ... View answer 6 votes The best tool to resolve (multi-) collinearity is in my view the Cholesky-decomposition of the correlation/covariance matrix. The following example discusses even the case of collinearity, where none ... View answer 6 votes Hmm, after I' done an example in my MatMate-language I see that there is already a python-answer, which might be preferable because python is widely used. But because you had still questions I show ... View answer Accepted answer 6 votes It seems to me that the following is the mathematically simplest way to partial-out variables from a correlated set of items. Consider a correlation matrix R for 5 items, where we want to "partial-... View answer 5 votes At the front side of the famous statistician's office:$$ \mathfrak{ \text{"Let's make the crowd a population" }\\ \text{Thus spoke the Lord and created the statistician.}} $$At the front ... View answer 4 votes I've in another question the following correlation matrix C for the three variables X,Y,Z given:$$ \text{ C =} \small \left[ \begin{array} {rrr} 1&-0.286122&-0.448535\\ -0.286122&1&...

This is not (yet) and answer, only a comment but too long for the box I do not really know how to determine such significance; but out of couriosity I did a bootstrap-procedure: from a ...

I think you're talking of "usefulness". In simplest terms it can be explained with the help of a cholesky-decomposition of the correlation/covariance-matrix, where the dependent variable is at ...

(This is really a comment to @ttnphns's second answer) As far as the different type of reproduction of covariance assuming error by PC and by FA is concerned, I've simply printed out the loadings/...

Well, the other answer gives an example, when the log-transform is used to reduce the influence of extreme values or outliers. Another general argument occurs, when you try to analyze data which are ...

As the determinant is zero we have only one factor. The variables are negatively correlated so their loadings must have different sign. So without checking the actual numbers, solution #1 is the only ...

There are PCA-like procedures for the so-called "oblique" case. In stat-software like SPSS (and possibly also in its freeware clone) PSPP one finds the equivalently called "oblique rotations", and ...

I computed $$\small b(r) = \sqrt{\small\text{comp}_1(r)^2 + \text{comp}_2(r)^2}$$ and $$\small \text{err}(r)= \small \text{comp}_{1\&2}(r) - b(r)$$ where $r$ indicates the row-index, so comp1&...

For me it was much helpful to read in the book of S. Mulaik "The Foundations of Factoranalysis" (1972), that there is a method purely of rotations of a matrix of factor loadings to arrive at a ...

Although this question has already an accepted answer I'd like to add something to the point of the question. "PCA" -if I recall correctly - means "principal components analysis"; so ...

The OP's question interests me on a more general level. In a lot of private studies over years I've done some experiments with a rotational scheme of estimating factors in the sense of CFA-definitions ...

This problem has often been discussed; one keyword is comparision of "centroid-rotation" vs. "pc-rotation", another one is "parcels" vs. "item-factor-analysis" (or so, I may not be up-to-date). The ...

Working with othogonal eigenvector matrices M (created as random rotation matrices) a sequence of experiments suggested, that "varimin"-rotation (which is just minimizing the same criterion which "...

General remarks: One should not varimax-rotate eigenvectors of PCA but loadings of PCA (i.e. eigenvectors scaled up by respective standard deviations). Also, it does not make sense to rotate all ...

Possibly this article (which is online readable) is also helpful. In the following the abstract: Abstract The maximum likelihood PCA (MLPCA) method has been devised in chemometrics as a ...

I've done this with an iterative optimum-search, and tested 2 versions. I've taken the original arrays and centered them calling this arrays cSRC and cTAR . Then I've done a loop with angles $\varphi$ ...

I want to illustrate the difference of the use of SS type I and SS type III as hinted by @ttnphns in his comment. This explains the different result gotten by the software and says also how to arrive ...

I don't know what's going on there. I've got with that R-commands the following result: Response: y Df Sum Sq Mean Sq F value Pr(>F) x1 1 0.156369 0.156369 101.7309 0....

As others have already shown: cholesky works. Here a piece of code which is very short and very near to pseudocode: a codepiece in MatMate: Co = {{1.0, 0.6, 0.9}, _ {0.6, 1.0, 0.5}, _ {...

Just one additional remark for @amoebas's long (and really great) answer on the character of the $\Psi$-estimate. In your initial statements you have three $\Psi$: for PCA is $\Psi = 0$, ...

Just to prevent to set correlation which are "impossible" as a whole set (the matrix of correlations can become non-positivedefinite) - for instance you can't define two nearly correlated variables ...

In my view, the notions of "PCA" and "FA" are on a different dimension from that of notions of "exploratory", "confirmatory" or maybe "inferential". So each of the two mathematical/statistical ...