adding a small constant to the diagonals of a matrix to stabilize I have a large correlation matrix (110x110) with some small eigenvalues (about 20 < 0.1). It has been suggested that adding a constant (about 0.1) to the diagonals will help to stabilize the matrix. My ultimate goal is to run dimensionality reduction on this matrix (PCA or Factor analysis). So when I do that, should I be using the "stable" matrix? How would one expect the results to change due to adding the constants?
 A: This is in addition to OmG's good answer (+1).
Numerically, the size of the eigenvalues that is not directly a problem but rather the condition number of the matrix (i.e. the ratio between the largest and the smallest singular value) as this is what it relates to the "stability" of the system. What you propose (adding a small constant along the diagonal of the covariance/correlation matrix) is effectively a ridge regression/regularisation solution. (I think it is actually a very good solution). Other options would be: 1. Get more data, 2. Eliminate certain explanatory variables or 3. combine some of the  explanatory variables to form new ones.
Regarding the potential influence the results of a PCA analysis might experience due to this regularisation step: Given the $110 \times 110$ initial correlation matrix $A$ you described, qualitatively we should see very small differences in the estimated principal components of the major orders (e.g. $k = \{1,...,5\}$, etc.) but we should expect more pronounced differences in the minor orders (e.g. $k = \{60+\}$). Notice that if we use a truncated SVD approach to do our PCA in our original sample $Y$ (not the correlation matrix $A$) we will never worry for those minor order principal components anyway. We would only care for the top $k$ components and the lower magnitude components that would be potentially affect by near-collinearity would be excluded from the analysis automatically. 
I recently read Spanos (2018) "Near-collinearity in linear regression revisited: The numerical vs. the statistical perspective" and I think it will greatly assist one's understanding about the possible implication of near-collinearity can have in an analysis (sign-reversal, changes in magnitude or loss of significance when testing, etc.) The paper takes the position of treating near-collinearity, "numerically" as a data problem (effectively what you describe when looking at eigenvalues, norms, etc.) and "statistically" as a parameter estimation problem. (effectively why we worry about it). (A free copy can be found here.)
A: Because the eigenvalues come from $det(A-\lambda I) = 0$. Hence, if you add some constant $c$ to the elements of the diagonal of $A$, you will have $det(A + cI - \lambda I) = det (A-(\lambda -c)I)$. Hence, the eigenvalue of the $A' = A + cI$, as $c$ is greater than $\lambda$ and $\lambda$ is small, is near to $c$ and the problem of small eigenvalues will be solved for $A'$.
