Condition number of covariance matrix

Question

I am interested in generating a covariance matrix of dimension say 100. I managed to get a correlation matrix with finite condition number.

To construct a covariance matrix I need to have standard deviations. I think for my case the most suitable one is to generate standard deviations from gamma distribution.

So, it gives me small standard deviations as well as large standard deviations. As a result of that, the resulting covariance matrix has a very high condition number.

I want to know whether the condition number can be affected by the scale of the variables and if I want to incorporate different scales in the covariance matrix how can I get a covariance matrix with a reasonable condition number?

Any help or insight regarding this is highly appreciated.

Answer 1

Yes, the scales of your variables affect the condition number. This is a real phenomenon with practical consequences; for example, I am using linear least-squares to solve a fitting problem, and if I just drop in the appropriate columns my condition number is of order 10^18 (presumably worse, as this is the limit of my numerical precision). If on the other hand I rescale my variables so each column of the fit matrix has the same sum-of-squares amplitude, the condition number of the fit matrix drops to less than a hundred. If I use the ill-conditioned matrix to compute fit values, they and the residuals are terrible; if I use the rescaled matrix and then rescale the variables, I get good stable fits.

What this means in terms of correlation and covariance matrices is that if you want to work with differently-scaled variables, you should keep the individual variable scales separate from the correlation matrix. If you do this, then a bad condition number of the correlation matrix corresponds to real, strong correlations between your variables. If you construct a covariance matrix by multiplying the scales in, then indeed, you can get a bad condition number just because your variables have different scales.

You don't say exactly what you want to do with your generated covariance matrices. If you're trying to evaluate the performance of an algorithm, then you have revealed a shortcoming in that algorithm: it works better if you rescale all your variables first. If you're doing something else, well, the fact is that if your variables have different scales, the covariance matrices really will have horrible condition numbers.

Answer 2

In general, it is really really unlikely the covariance matrix is ill-conditioned. There are results by Tao and Vu (http://arxiv.org/pdf/math/0703307v1.pdf theorem P2). General rule I keep in mind is Marcenko-Pastur: If you have each column of a matrix X of dimension N*P being sampled independently then so long as (N/P) or (P/N) is not close to 1 you will not get ill-conditioning. (i.e. as a rule of thumb, you are generally safe if you multiply 2 matrices as $EE^{T}$ where the dimensions are not close to one another. This is the case I frequently encounter)

Besides, if you know the spectrum of the correlation matrix, the answer is known analytically.

Write the Cholesky-decomposition of the correlation matrix

$C = GG^{T}$

The Covariance matrix will be

$S = \Sigma GG^{T} \Sigma$ where $\Sigma$ is a diagonal matrix having standard deviations.

Therefore, the condition number of $S$ is the square of the condition number of $\Sigma G$ which you can find exactly if you so desire

Answer 3

Why don't you draw your covariance matrix from an inverse Wishart distribution? Gamma distribution is usually used as a prior for a single dimensional variance, Wishart is the multivariate case of the Gamma distribution. It is used as the conjugate prior for the covariance of a multi-variate normal. Sampling the values on the diagonal and the off-diagonal values separately actually does not make much sense, since these are dependent, right?

There are built-in functions (for Matlab, Python etc...) to draw from the inverse Wishart and you supply it with a positive definite matrix as the scale parameter, so condition number should not be a problem for the drawn samples.

Answer 4

Easiest to interpret is to generate a spectrum and the orthogonal group (rotation matrix): $V^T D V$. You can put whatever prior you want on the eigenvalues. Probably there are some good ones depending on context.