My data consists of approx. 5 Million binary strings (n) and every string is 2788 characters long. My goal is to find out if position i is correlated with position j. I approximated a covariance matrix the following way:
P(Xi=1) := (C(i) + 0.5) / (n + 1) = E(i)
C(i) := number of strings with a 1 at position 1
P(Xi = 1, Xj = 1) := (C11 + 0.5) / (n + 2) = p11
Ckl := number of strings with position i = k and position j = l
-> for example C11 := number of strings with position i = 1 and position j = 1
P(Xi=0), P(Xi = 0, Xj = 0) and so on are defined equivalently I used pseudocounts, because no possibility is allowed to be exactly one or zero, because I know that is not possible according to my data.
Cov(Xi,Xj) = p11 * (1 - P(Xi=1)) * (1 - P(Xj=1)) + p01 * (- P(Xi=0)) * (1 - P(Xj=1)) + p10 * (1 - P(Xi=1)) * (- P(Xj=1)) + p00 * (- P(Xi=0)) * (- P(Xj=1))
Now, I want to calculate an inverse covariance matrix. For calculating the covariance matrix, I use the package QUIC As soon as I have the inverse covariance matrix, I want to generate a graph representing the correlations between the i-th and j-th random variables, so I am only interested in non-zero entries who are not on the diagonal.
But I don't know how I should choose the regularization parameter... if I choose 1 as regularization parameter, all entries of the inverse covariance matrix are zero, expect the ones on the diagonal. Thats bad, so I also tried several other values for the parameter and it worked. Now, I have differnet inverse covariance matrixes, but I don't know how to find out which one fits my data best.
I have also thought about using cross-validation, for example a k-fold-cross-validation. There is no problem in dividing my dataset into different pieces and generateing the inverse covariance matrix, but I don't know how to calculate an error for matrices... is it even possible?
And sorry if you might think that my question is dumb or something. I have only little expierience in the field of statistics.