Gottfried Helms
  • Member for 11 years, 2 months
  • Last seen this week
Is PCA unstable under multicollinearity?
Accepted answer
13 votes

The answer might be given in even simpler terms: the multiple regression has one step more than the pca if seen in terms of linear algebra, and from the second step the instability comes into ...

View answer
Partial Correlation Interpretation
Accepted answer
10 votes

For understanding this I always prefer the cholesky-decomposition of the correlation-matrix. Assume the correlation-matrix R of the three variable $X.Y.Z$ as $$ \text{ R =} \left[ \begin{array} {rrr} ...

View answer
"matrix is not positive definite" - even when highly correlated variables are removed
6 votes

The best tool to resolve (multi-) collinearity is in my view the Cholesky-decomposition of the correlation/covariance matrix. The following example discusses even the case of collinearity, where none ...

View answer
How to generate a large full-rank random correlation matrix with some strong correlations present?
6 votes

Hmm, after I' done an example in my MatMate-language I see that there is already a python-answer, which might be preferable because python is widely used. But because you had still questions I show ...

View answer
How to "regress out" some variables?
Accepted answer
6 votes

It seems to me that the following is the mathematically simplest way to partial-out variables from a correlated set of items. Consider a correlation matrix R for 5 items, where we want to "partial-...

View answer
Statistics Jokes
5 votes

At the front side of the famous statistician's office: $$ \mathfrak{ \text{"Let's make the crowd a population" }\\ \text{Thus spoke the Lord and created the statistician.}} $$ At the front ...

View answer
Are standardized betas in multiple linear regression partial correlations?
4 votes

I've in another question the following correlation matrix C for the three variables X,Y,Z given: $$ \text{ C =} \small \left[ \begin{array} {rrr} 1&-0.286122&-0.448535\\ -0.286122&1&...

View answer
How to identify variables with significant loadings in PCA?
Accepted answer
4 votes

This is not (yet) and answer, only a comment but too long for the box I do not really know how to determine such significance; but out of couriosity I did a bootstrap-procedure: from a ...

View answer
Looking for name of statistical property (variance explained by regressor in addition to all other regressors)
Accepted answer
4 votes

I think you're talking of "usefulness". In simplest terms it can be explained with the help of a cholesky-decomposition of the correlation/covariance-matrix, where the dependent variable is at ...

View answer
Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis?
3 votes

(This is really a comment to @ttnphns's second answer) As far as the different type of reproduction of covariance assuming error by PC and by FA is concerned, I've simply printed out the loadings/...

View answer
Why log-transforming the data before performing principal component analysis?
3 votes

Well, the other answer gives an example, when the log-transform is used to reduce the influence of extreme values or outliers. Another general argument occurs, when you try to analyze data which are ...

View answer
Identifying the coefficients of a principal component
Accepted answer
3 votes

As the determinant is zero we have only one factor. The variables are negatively correlated so their loadings must have different sign. So without checking the actual numbers, solution #1 is the only ...

View answer
Non-orthogonal technique analogous to PCA
3 votes

There are PCA-like procedures for the so-called "oblique" case. In stat-software like SPSS (and possibly also in its freeware clone) PSPP one finds the equivalently called "oblique rotations", and ...

View answer
When doing PCA, is it possible or correct to find the correlation between variables and the first two components combined?
Accepted answer
2 votes

I computed $$\small b(r) = \sqrt{\small\text{comp}_1(r)^2 + \text{comp}_2(r)^2} $$ and $$\small \text{err}(r)= \small \text{comp}_{1\&2}(r) - b(r) $$ where $r$ indicates the row-index, so comp1&...

View answer
How to visualize what canonical correlation analysis does (in comparison to what principal component analysis does)?
2 votes

For me it was much helpful to read in the book of S. Mulaik "The Foundations of Factoranalysis" (1972), that there is a method purely of rotations of a matrix of factor loadings to arrive at a ...

View answer
Is PCA followed by a rotation (such as varimax) still PCA?
2 votes

Although this question has already an accepted answer I'd like to add something to the point of the question. "PCA" -if I recall correctly - means "principal components analysis"; so ...

View answer
Factor expressed as standardised variable in factor analysis
1 votes

The OP's question interests me on a more general level. In a lot of private studies over years I've done some experiments with a rotational scheme of estimating factors in the sense of CFA-definitions ...

View answer
Comparing the first principal component with an observed variable (mean)?
1 votes

This problem has often been discussed; one keyword is comparision of "centroid-rotation" vs. "pc-rotation", another one is "parcels" vs. "item-factor-analysis" (or so, I may not be up-to-date). The ...

View answer
How is the proof that the Quartimax/Varimax-rotation converges?
Accepted answer
1 votes

Working with othogonal eigenvector matrices M (created as random rotation matrices) a sequence of experiments suggested, that "varimin"-rotation (which is just minimizing the same criterion which "...

View answer
Why does varimax applied to PCA outcome fail to do anything at all?
Accepted answer
1 votes

General remarks: One should not varimax-rotate eigenvectors of PCA but loadings of PCA (i.e. eigenvectors scaled up by respective standard deviations). Also, it does not make sense to rotate all ...

View answer
What is maximum likelihood PCA?
1 votes

Possibly this article (which is online readable) is also helpful. In the following the abstract: Abstract The maximum likelihood PCA (MLPCA) method has been devised in chemometrics as a ...

View answer
Find the rotation between set of points
1 votes

I've done this with an iterative optimum-search, and tested 2 versions. I've taken the original arrays and centered them calling this arrays cSRC and cTAR . Then I've done a loop with angles $\varphi$ ...

View answer
Anova of metric items: SPSS and R display different square-sums and F-values. Which is the better philosophy?
1 votes

I want to illustrate the difference of the use of SS type I and SS type III as hinted by @ttnphns in his comment. This explains the different result gotten by the software and says also how to arrive ...

View answer
Partition of sums of squares (ANOVA)
1 votes

I don't know what's going on there. I've got with that R-commands the following result: Response: y Df Sum Sq Mean Sq F value Pr(>F) x1 1 0.156369 0.156369 101.7309 0....

View answer
How to use the Cholesky decomposition, or an alternative, for correlated data simulation
1 votes

As others have already shown: cholesky works. Here a piece of code which is very short and very near to pseudocode: a codepiece in MatMate: Co = {{1.0, 0.6, 0.9}, _ {0.6, 1.0, 0.5}, _ {...

View answer
Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis?
1 votes

Just one additional remark for @amoebas's long (and really great) answer on the character of the $\Psi$-estimate. In your initial statements you have three $\Psi$: for PCA is $ \Psi = 0$, ...

View answer
Tool for generating correlated data sets
1 votes

Just to prevent to set correlation which are "impossible" as a whole set (the matrix of correlations can become non-positivedefinite) - for instance you can't define two nearly correlated variables ...

View answer
Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis?
1 votes

In my view, the notions of "PCA" and "FA" are on a different dimension from that of notions of "exploratory", "confirmatory" or maybe "inferential". So each of the two mathematical/statistical ...

View answer
Exploratory factor analysis with fixed factors
1 votes

(This is possibly a comment rather than an answer but is long) To assume a fixed number of factors a-priori usually means to be on the side of testing/confirming hypotheses, for instance by a ...

View answer
Is there a statistical justification for removing items from a scale with good reliability?
1 votes

I think, that the “ancient” argumentation with computing complexity is obsolete today with the fast computers. However, it might be of interest to reduce the number of items to reduce the burden of ...

View answer