# Condition number of covariance matrix

I am interested in generating a covariance matrix of dimension say 100. I managed to get a correlation matrix with finite condition number.

To construct a covariance matrix I need to have standard deviations. I think for my case the most suitable one is to generate standard deviations from gamma distribution.

So, it gives me small standard deviations as well as large standard deviations. As a result of that, the resulting covariance matrix has a very high condition number.

I want to know whether the condition number can be affected by the scale of the variables and if I want to incorporate different scales in the covariance matrix how can I get a covariance matrix with a reasonable condition number?

Any help or insight regarding this is highly appreciated.

Yes, the scales of your variables affect the condition number. This is a real phenomenon with practical consequences; for example, I am using linear least-squares to solve a fitting problem, and if I just drop in the appropriate columns my condition number is of order 10^18 (presumably worse, as this is the limit of my numerical precision). If on the other hand I rescale my variables so each column of the fit matrix has the same sum-of-squares amplitude, the condition number of the fit matrix drops to less than a hundred. If I use the ill-conditioned matrix to compute fit values, they and the residuals are terrible; if I use the rescaled matrix and then rescale the variables, I get good stable fits.

What this means in terms of correlation and covariance matrices is that if you want to work with differently-scaled variables, you should keep the individual variable scales separate from the correlation matrix. If you do this, then a bad condition number of the correlation matrix corresponds to real, strong correlations between your variables. If you construct a covariance matrix by multiplying the scales in, then indeed, you can get a bad condition number just because your variables have different scales.

You don't say exactly what you want to do with your generated covariance matrices. If you're trying to evaluate the performance of an algorithm, then you have revealed a shortcoming in that algorithm: it works better if you rescale all your variables first. If you're doing something else, well, the fact is that if your variables have different scales, the covariance matrices really will have horrible condition numbers.

In general, it is really really unlikely the covariance matrix is ill-conditioned. There are results by Tao and Vu (http://arxiv.org/pdf/math/0703307v1.pdf theorem P2). General rule I keep in mind is Marcenko-Pastur: If you have each column of a matrix X of dimension N*P being sampled independently then so long as (N/P) or (P/N) is not close to 1 you will not get ill-conditioning. (i.e. as a rule of thumb, you are generally safe if you multiply 2 matrices as $EE^{T}$ where the dimensions are not close to one another. This is the case I frequently encounter)

Besides, if you know the spectrum of the correlation matrix, the answer is known analytically.

Write the Cholesky-decomposition of the correlation matrix

$C = GG^{T}$

The Covariance matrix will be

$S = \Sigma GG^{T} \Sigma$ where $\Sigma$ is a diagonal matrix having standard deviations.

Therefore, the condition number of $S$ is the square of the condition number of $\Sigma G$ which you can find exactly if you so desire

• I downvoted because I think this answer is too short for it's own good and gives a false sense of assurance. I have seen numerous covariance matrices being ill-conditioned; especially when real data are involved it is far from uncommon to have 0 eigenvalues due to finite precision arithmetic in large covariance matrices. Additionally the paper linked talks about matrices whose entries are that IID RV, $N(0,1)$; the OP clearly states he is interested on $\Gamma$ distributions; generalizing from the paper's results is far from obvious. – usεr11852 May 8 '15 at 4:51
• I have also seen covariance matrices with small condition numbers. That is irrelevant considering the reader has seen a well-conditioned (which is what I think he means by finite) correlation matrix. My note was conditioning on his observation. Marcenko-Pastur gives a very general framework to regulate condition numbers of covariance matrices (though the result is asymptotic, it has been shown to work extremely well for large samples. Given a condition number one desires and a correlation matrix, back-tracking becomes a lot easier – Sid May 8 '15 at 5:16
• I appreciate the time replying. Having said that you write: "In general" and not Given you have a finite condition number to your correlation matrix. To that extend the OP specifically comments about "small standard deviations as well as large standard deviations", ie. eigenvalues of different magnitudes which renders the findings of the Tao and Vu paper inapplicable (directly at least). I will have not problem retracting my vote (and possibly upvoting) if you elaborate on how Marcenko-Pastur would solve the OP's issue in your post. – usεr11852 May 8 '15 at 5:36
• Thanks for the help. I realize I had linked the wrong paper. The truth is the result holds 'in general'. It is unbelievably unlikely you have a random $N*N$ matrix with poor condition number (See theorem P2 of the new Tao and Vu link), let alone symmetric positive definite ones. Hence moral of the story: if I do see an ill-conditioned matrix, I generally mistrust something about its computation – Sid May 8 '15 at 5:41
• I still believe that the really really unlikely is a bit strong but OK, that's a minor thing. For me the post as it stands it is useful. (+1) – usεr11852 May 8 '15 at 6:15

Why don't you draw your covariance matrix from an inverse Wishart distribution? Gamma distribution is usually used as a prior for a single dimensional variance, Wishart is the multivariate case of the Gamma distribution. It is used as the conjugate prior for the covariance of a multi-variate normal. Sampling the values on the diagonal and the off-diagonal values separately actually does not make much sense, since these are dependent, right?

There are built-in functions (for Matlab, Python etc...) to draw from the inverse Wishart and you supply it with a positive definite matrix as the scale parameter, so condition number should not be a problem for the drawn samples.

Easiest to interpret is to generate a spectrum and the orthogonal group (rotation matrix): $$V^T D V$$. You can put whatever prior you want on the eigenvalues. Probably there are some good ones depending on context.