5
$\begingroup$

For the purpose of dimension reduction I have performed an eigen analysis (using Jacobi-iteration) on a correlation matrix R of 163 variables (based on 1500 cases). The scree plot is attached. Scree-plot

The sum of all eigenvalues is (to 6 places after the decimal) equal to the number of variables, as expected. However, the last 54 eigenvalues are negative, and the cumulative explained variance goes to 125% before droping to 100%. What is going on here, and does it still make sense to use the first 15 or so eigenvectors to calculate scores and communalities?

There are missing data in the set, therefore a robust calculation of correlation coefficients was used:
set sums and n to 0
for i := 1 to Cases do
if ((IsNaN(x[i]) or (IsNaN(y[i]))
then do nothing
else add $x, y, xy, x^2, y^2$ to their respective sums and inc(n)
$r = \frac{n * SumXY - SumX * SumY}{\sqrt{[n * SumX^2 - (SumX)^2] [n* SumY^2 - (SumY)^2]}}$ The idea is to use the available information completely without inventing anything by imputation (at this state, for the calculation of scores by multiplication of data and eigenvectors I see no way other than imputation of $\bar{x}$).

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ Something is wrong here. Without you sharing the data and/or the code it will be hard to say what. $\endgroup$
    – amoeba
    Commented Nov 23, 2017 at 13:58
  • 3
    $\begingroup$ This scree plot proves that either (a) the PCA software is flawed or (much more likely, if it's not home-grown software) (b) your matrix is not a true correlation matrix. The usual cause of this is computing pairwise correlations when there are missing values. All these issues have been discussed in other threads: use keywords like "PCA", "covariance", "correlation", "eigenvalue", and "definite" to find them. $\endgroup$
    – whuber ♦
    Commented Nov 23, 2017 at 14:05
  • 1
    $\begingroup$ Yes, there are missing values. However, I could not find any previous answers on this with the keywords you suggested (or any I had thought of myself) $\endgroup$
    – Engelbert Buxbaum
    Commented Nov 24, 2017 at 13:38
  • 1
    $\begingroup$ So how did you compute the correlation matrix given that you have missing values in the data? If you ever reply, please include @amoeba into your reply. $\endgroup$
    – amoeba
    Commented Nov 28, 2017 at 14:17
  • 2
    $\begingroup$ Your output is strong proof that your claim is incorrect: any "robust" estimation of the correlation matrix will necessarily not produce large negative eigenvalues! There seem to be a couple of relevant posts found in this search: stats.stackexchange.com/… $\endgroup$
    – whuber ♦
    Commented Dec 1, 2017 at 19:24

2 Answers 2

Reset to default
4
$\begingroup$

If you calculate pairwise correlation coefficients in presence of missing values, you correlation matrix may end up being non positive definite. In fact it's a very common phenomenon in quant finance. One way to deal with this issue is Ledoit Wolf procedure, see here. They developed a method for a different issue, but it's used for missing value issue too. An author has MATLAB code here.

Suppose you have three variables x,y and z. In observation 1 value of x is missing, but y and z are present. One way to calculate the correlation matrix is to skip the observation 1.

Another, seemingly better way is to skip observation 1 only when calculating pairwise correlations xy and xz, and use it for YZ correlation. Values of Y and Z are available in observation 1, why not use them? If you go this way then obtained correlation matrix may not be a good estimate of the true correlation matrix, surprisingly. Particularly, your matrix may not be positive definite. Again, this is a common situation in many finance applications such as portfolio optimization and PCA.

I would skip the observations with missing values IF the data size allows it. This is not always possible, e.g. sometimes we have hundreds variables and about as many observations. If we skip an observation when at least one variable value is missing, easily a half of the observations may be flagged. In this case, it's worth the trouble to do pairwise correlations using "all available" data, then shrink the matrix using Ledoit Wolf procedure. Otherwise, if it was just a couple of rows dropping out, then I wouldn't bother and skip them.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Thank you for the Ledoit-Wolf procedure. I picked up on that reference because the approach looks like it might solve a problem I posted long ago at stats.stackexchange.com/questions/1781. Is that a realistic assessment? That is, should I spend some time studying the possible extension of that procedure to the problem of estimating a covariance matrix from censored data--or do you think this would be unlikely to work out? $\endgroup$
    – whuber ♦
    Commented Dec 1, 2017 at 20:52
  • 2
    $\begingroup$ @whuber I didn't see your question before. It seems that Ledoit Wolf approach is worth trying in this case. I used it in quant finance apps, where some assets are rarely traded and have a lot of missing observations, more than 50%. I also used it in cases where some series are much shorter than the others. In this case covariance matrix can easily become non PD. $\endgroup$
    – Aksakal
    Commented Dec 1, 2017 at 21:44
  • $\begingroup$ @Aksakal: Thanks for this reference, I check it out and report back what happens. Just skipping observations isn't an option, with 163 variables even 10% missing data wouldn't leave many observations to perform PCA on. In addition, these are student performance data, and I wish to predict from admission data the final success. If students failing along the course are removed from the dataset, I'd get only what successful students have in common, but not what distinguishes them from the not so successful. $\endgroup$
    – Engelbert Buxbaum
    Commented Dec 2, 2017 at 15:06
  • $\begingroup$ Ok, I (kind of) see. The trick is to move the eigenvalues towards their grand mean, while leaving the eigenvectors intact. A more readable paper may be epublications.bond.edu.au/cgi/…, and a more modern approach can be found in poseidon01.ssrn.com/… $\endgroup$
    – Engelbert Buxbaum
    Commented Dec 3, 2017 at 9:57
  • $\begingroup$ Nice, I didn’t know they came up with a new method, will try this one too $\endgroup$
    – Aksakal
    Commented Dec 3, 2017 at 15:43
2
$\begingroup$

After the discussion here, I can provide at least a partial answer. Apparently, pairwise calculation of correlation coefficients, especially if the data matrix has missing data, leads to a correlation matrix that is only positive semi-definite, not positive definite. Eigenvalues that are too extreme. According to http://epublications.bond.edu.au/cgi/viewcontent.cgi?article=1099&context=ejsie it is possible to counteract this effect by "shrinking" the matrix. To this purpose, a weighted average of the correlation matrix $R$ with a form matrix $F$ is calculated: $\hat{r}_{ij} = (1-\omega) r_{ij} + \omega f_{ij}$ with $\omega$ a weight factor from [0..1] (in the literature, it is often called $\lambda$, but as this symbol is used for eigenvalues already I have re-christend it). According to https://cssanalytics.files.wordpress.com/2013/10/shrinkage-simpler-is-better.pdf it doesn't really matter which form matrix one uses, I have tried both the identity matrix $I$ and the average matrix $\bar{R}$ (for each variable $i$ calculate the average correlation with all other variables $\bar{r}_i$, then $\bar{r}_{ij} = (\bar{r}_i + \bar{r}_j)/2$). The results are indeed very similar.

What I still don't understand is why $R$ is so affected by missing data. I have done simulations with 10,000 data from $y = 1 + 2x + \epsilon$, with $\epsilon$ a Gaussian random number so that $r = 0.825$. Then I have set random elements of this matrix to NaN and recalculated $r$. For up to 20% missing data, the maximal deviation of $r$ was 0.01, and most of the data fell within ± 0.005. Even for 50% missing data the maximum deviation was 0.015, for 70% 0.02. Surely this is not such a big effect?

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ (1) Pairwise calculation of correlation coefficients does not necessarily lead to positive semi-definite matrices: it can create practically any symmetric matrix (with positive entries on the diagonal, of course), even indefinite ones. (2) Accordingly, the eigenvalues are not necessarily extreme at all. The shrinking appears to be an ad hoc procedure to eliminate negative eigenvalues. (3) The effect will be largest with small datasets where the missingness is not random and the original eigenvalues are well away from zero. $\endgroup$
    – whuber ♦
    Commented Dec 10, 2017 at 17:04
  • $\begingroup$ @whuber, my experiments with shrinkage lead to the new matrix with completely different from the original ones eigen values. As a result, PCA I'd like to apply to the new matrix is going to be nonsense. Is there way to preserve magnitude of at least first few largest eigen values after the shrinkage procedure? $\endgroup$
    – Stepan S. Sushko
    Commented Aug 30, 2023 at 12:31
  • $\begingroup$ @Stepan I suspect the limited space available for comments made it difficult for you to describe your problem in enough detail to respond. Consider, then, posting that as a question in a new thread where you can give a fuller explanation of what you're doing and what your objective is. $\endgroup$
    – whuber ♦
    Commented Aug 30, 2023 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.