2
$\begingroup$

My data consists of approx. 5 Million binary strings (n) and every string is 2788 characters long. My goal is to find out if position i is correlated with position j. I approximated a covariance matrix the following way:

P(Xi=1) := (C(i) + 0.5) / (n + 1) = E(i)

C(i) := number of strings with a 1 at position 1

P(Xi = 1, Xj = 1) := (C11 + 0.5) / (n + 2) = p11

Ckl := number of strings with position i = k and position j = l

-> for example C11 := number of strings with position i = 1 and position j = 1

P(Xi=0), P(Xi = 0, Xj = 0) and so on are defined equivalently I used pseudocounts, because no possibility is allowed to be exactly one or zero, because I know that is not possible according to my data.

Cov(Xi,Xj) = p11 * (1 - P(Xi=1)) * (1 - P(Xj=1)) + p01 * (- P(Xi=0)) * (1 - P(Xj=1)) + p10 * (1 - P(Xi=1)) * (- P(Xj=1)) + p00 * (- P(Xi=0)) * (- P(Xj=1))

Now, I want to calculate an inverse covariance matrix. For calculating the covariance matrix, I use the package QUIC As soon as I have the inverse covariance matrix, I want to generate a graph representing the correlations between the i-th and j-th random variables, so I am only interested in non-zero entries who are not on the diagonal.

But I don't know how I should choose the regularization parameter... if I choose 1 as regularization parameter, all entries of the inverse covariance matrix are zero, expect the ones on the diagonal. Thats bad, so I also tried several other values for the parameter and it worked. Now, I have differnet inverse covariance matrixes, but I don't know how to find out which one fits my data best.

I have also thought about using cross-validation, for example a k-fold-cross-validation. There is no problem in dividing my dataset into different pieces and generateing the inverse covariance matrix, but I don't know how to calculate an error for matrices... is it even possible?

And sorry if you might think that my question is dumb or something. I have only little expierience in the field of statistics.

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ "I have a covariance matrix and want to calculate an inverse covariance matrix." If you already know the covariance matrix, why are you trying to estimate the inverse? Have you tried simply computing the inverse directly? $\endgroup$
    – JohnA
    Commented May 29, 2015 at 15:35
  • $\begingroup$ Because I was told that it is not possible for large problems. My covariance matrix has 2788 x 2788 non-zero entries. $\endgroup$
    – black.firefly
    Commented May 29, 2015 at 16:51
  • $\begingroup$ I'm a little unsure what you're trying to do. The covariance matrix almost directly gives you the "correlations between the random variables" without computing an inverse at all. Perhaps you are trying to compute partial correlation coefficients? With so many variables it certainly is a challenging problem (in terms of stability and resistance to roundoff error). Another line of attack would be to investigate whether you really need all 2788 variables. Can you share any thoughts about this? $\endgroup$
    – whuber ♦
    Commented May 29, 2015 at 17:20
  • $\begingroup$ @black.firefly It's certainly possible to compute the inverse of a 3k x 3k matrix, though it may take a little time. If you don't need to do this repeatedly and quickly, just be sure you're using a fast linear algebra library like MKL or OpenBLAS and give it some time. $\endgroup$
    – Danica
    Commented May 29, 2015 at 17:21
  • 1
    $\begingroup$ No apologies needed - if you can go back and edit your question to include more information on your data and how you went about estimating the covariance matrix initially, we might be able to offer more specific input. $\endgroup$
    – JohnA
    Commented Jun 1, 2015 at 14:29

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.