# Adding a small constant to the diagonals of a matrix to stabilize

I have a large correlation matrix (110x110) with some small eigenvalues (about 20 < 0.1). It has been suggested that adding a constant (about 0.1) to the diagonals will help to stabilize the matrix. My ultimate goal is to run dimensionality reduction on this matrix (PCA or Factor analysis). So when I do that, should I be using the "stable" matrix? How would one expect the results to change due to adding the constants?

Numerically, the size of the eigenvalues that is not directly a problem but rather the condition number of the matrix (i.e. the ratio between the largest and the smallest singular value) as this is what it relates to the "stability" of the system. What you propose (adding a small constant along the diagonal of the covariance/correlation matrix) is effectively a ridge regression/regularisation solution. (I think it is actually a very good solution). Other options would be: 1. Get more data, 2. Eliminate certain explanatory variables or 3. combine some of the explanatory variables to form new ones.

Regarding the potential influence the results of a PCA analysis might experience due to this regularisation step: Given the $$110 \times 110$$ initial correlation matrix $$A$$ you described, qualitatively we should see very small differences in the estimated principal components of the major orders (e.g. $$k = \{1,...,5\}$$, etc.) but we should expect more pronounced differences in the minor orders (e.g. $$k = \{60+\}$$). Notice that if we use a truncated SVD approach to do our PCA in our original sample $$Y$$ (not the correlation matrix $$A$$) we will never worry for those minor order principal components anyway. We would only care for the top $$k$$ components and the lower magnitude components that would be potentially affect by near-collinearity would be excluded from the analysis automatically.

I recently read Spanos (2018) "Near-collinearity in linear regression revisited: The numerical vs. the statistical perspective" and I think it will greatly assist one's understanding about the possible implication of near-collinearity can have in an analysis (sign-reversal, changes in magnitude or loss of significance when testing, etc.) The paper takes the position of treating near-collinearity, "numerically" as a data problem (effectively what you describe when looking at eigenvalues, norms, etc.) and "statistically" as a parameter estimation problem. (effectively why we worry about it). (A free copy can be found here.)

• Thanks! This is very useful. I will look up the reference. Commented Feb 3, 2019 at 17:53

Because the eigenvalues come from $$\operatorname{det}(A-\lambda I) = 0$$. Hence, if you add some constant $$c$$ to the elements of the diagonal of $$A$$, you will have $$\operatorname{det}(A + cI - \lambda I) = \operatorname{det}(A-(\lambda -c)I)$$. Hence, the eigenvalue of the $$A' = A + cI$$, as $$c$$ is greater than $$\lambda$$ and $$\lambda$$ is small, is near to $$c$$ and the problem of small eigenvalues will be solved for $$A'$$.

• This is correct but incomplete, because "stability" here must refer to numerical stability, which depends on the condition number of the matrix, but not on the smallest eigenvalue alone. Thus, the final step in the analysis should be to point out that the ratio of largest to smallest eigenvalue in a PSD matrix decreases (quite rapidly) when a small positive multiple $c$ of the identity matrix is added. Even better, that gives practical quantitative information concerning how large to make $c$ to reduce the condition number to any desired level.
– whuber
Commented Apr 8 at 13:52