My data consists of approx. 5 Million binary strings (n) and every string is 2788 characters long. My goal is to find out if position i is correlated with position j. I approximated a covariance matrix the following way:

P(Xi=1) := (C(i) + 0.5) / (n + 1) = E(i)

C(i) := number of strings with a 1 at position 1

P(Xi = 1, Xj = 1) := (C11 + 0.5) / (n + 2) = p11

Ckl := number of strings with position i = k and position j = l

-> for example C11 := number of strings with position i = 1 and position j = 1

P(Xi=0), P(Xi = 0, Xj = 0) and so on are defined equivalently I used pseudocounts, because no possibility is allowed to be exactly one or zero, because I know that is not possible according to my data.

Cov(Xi,Xj) = p11 * (1 - P(Xi=1)) * (1 - P(Xj=1)) + p01 * (- P(Xi=0)) * (1 - P(Xj=1)) + p10 * (1 - P(Xi=1)) * (- P(Xj=1)) + p00 * (- P(Xi=0)) * (- P(Xj=1))

Now, I want to calculate an inverse covariance matrix. For calculating the covariance matrix, I use the package QUIC As soon as I have the inverse covariance matrix, I want to generate a graph representing the correlations between the i-th and j-th random variables, so I am only interested in non-zero entries who are not on the diagonal.

But I don't know how I should choose the regularization parameter... if I choose 1 as regularization parameter, all entries of the inverse covariance matrix are zero, expect the ones on the diagonal. Thats bad, so I also tried several other values for the parameter and it worked. Now, I have differnet inverse covariance matrixes, but I don't know how to find out which one fits my data best.

I have also thought about using cross-validation, for example a k-fold-cross-validation. There is no problem in dividing my dataset into different pieces and generateing the inverse covariance matrix, but I don't know how to calculate an error for matrices... is it even possible?

And sorry if you might think that my question is dumb or something. I have only little expierience in the field of statistics.

  • 2
    $\begingroup$ "I have a covariance matrix and want to calculate an inverse covariance matrix." If you already know the covariance matrix, why are you trying to estimate the inverse? Have you tried simply computing the inverse directly? $\endgroup$ – JohnA May 29 '15 at 15:35
  • $\begingroup$ Because I was told that it is not possible for large problems. My covariance matrix has 2788 x 2788 non-zero entries. $\endgroup$ – black.firefly May 29 '15 at 16:51
  • $\begingroup$ I'm a little unsure what you're trying to do. The covariance matrix almost directly gives you the "correlations between the random variables" without computing an inverse at all. Perhaps you are trying to compute partial correlation coefficients? With so many variables it certainly is a challenging problem (in terms of stability and resistance to roundoff error). Another line of attack would be to investigate whether you really need all 2788 variables. Can you share any thoughts about this? $\endgroup$ – whuber May 29 '15 at 17:20
  • $\begingroup$ @black.firefly It's certainly possible to compute the inverse of a 3k x 3k matrix, though it may take a little time. If you don't need to do this repeatedly and quickly, just be sure you're using a fast linear algebra library like MKL or OpenBLAS and give it some time. $\endgroup$ – Dougal May 29 '15 at 17:21
  • 1
    $\begingroup$ No apologies needed - if you can go back and edit your question to include more information on your data and how you went about estimating the covariance matrix initially, we might be able to offer more specific input. $\endgroup$ – JohnA Jun 1 '15 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.