5
$\begingroup$

A zero entry in the precision matrix (the inverse of the covariance matrix) means the corresponding variables are independent given all the other variables. For real-world data samples, when is an entry in the precision matrix small enough to be treated as a zero?

In my data-sample, if I adjust the precision matrix so all values< 0.004 are zero, the corresponding correlations do not change significantly. I got tho this value by trial and error: setting the threshold to 0.005 does cause significant changes in correlations.

The threshold value for precision matrix entries, below which variables can be considered conditionally independent, depends on sample size, on the number of variables (the size of the matrix) and on the other values in the matrix. Is there any way other than trial and error to find it?

Improve this question
$\endgroup$
1
  • $\begingroup$ you introduced a new tag precision-matrix can you please write a tag wiki? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Nov 10, 2017 at 10:26

1 Answer 1

Reset to default
3
$\begingroup$

Finding the covariance matrix that fits the data and has a conveniently large number of zero entries in it's inverse matrix is known as Covariance Selection (1). Zeros in the inverse covariance matrix are desirable both for computational and conceptual reasons: they indicate conditional independence between variables, making the model smaller.

Covariance selection is an active field of research with applications in domains from proteomics to economics. Several algorithms have been proposed and implementations are available in statistical software, for example glasso and smac in R. This presentation (2) provides a really nice overview.

Setting small values to zero does the same thing, but the glasso algorithm finds more zeros. For my particular problem of only 9 variables i could change 3 (of 36) connections to 0, while glasso found 9.

1) Dempster, A. P. Covariance Selection Biometrics, 1972, 28, 157-175

2) P. Olsen, F. Oztoprak, J. Nocedal and S. Rennie; Sparse Inverse Covariance Estimation; Summer Tutorial at IBM TJ Watson Research Center 2012

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ I've never used this procedure myself, but I'd be concerned that by setting some values in $\Sigma^{-1}$ to $0$, the covariance matrix $\Sigma$ would no longer be positive definite, and therefore would not be invertable, and breaks the relationship between covariance and precision matrices. Does your research shed any light on this problem? $\endgroup$
    – Sycorax ♦
    Commented Aug 27, 2015 at 14:13
  • $\begingroup$ Unfortunately it's trail-and-error: i set a threshold below which values are turned to 0, then i invert the matrix back, recalculate the correlation matrix and check if the differences are acceptable. This ensures a PD matrix, but finds less conditional independencies than better algorithms. I whish i understood the stability of matrix inversion better, i asked a related question about that: stats.stackexchange.com/questions/145412/… $\endgroup$
    – Ivana
    Commented Aug 28, 2015 at 11:27
  • $\begingroup$ The first sentence of the abstract (Dempster, 1972, Summary) is misleading: "The covariance structure of a multivariate normal population can be simplified by setting elements of the inverse of the covariance matrix to zero." With respect to the demonstrated algorithm in (Dempster, 1972, § 3), the widely repeated assertion that “covariance selection” inserts zeros in a precision matrix is false. Non-zero entries are placed in a precision matrix as covariance constraints are added to a maximum entropy distribution. See the appendix of scitepress.org/papers/2018/66446/66446.pdf $\endgroup$
    – krkeane
    Commented Apr 4, 2022 at 21:13
  • $\begingroup$ I'd like to re-emphasize that Dempster 1972 Summary is misleading. In section three, zeros are NOT inserted in the precision matrix. The exact output of Dempster section 3, plus corresponding precision matrices appears in linked paper above. Non-zeros are added to the precision matrix to impose constraints on pairwise covariance statistics. Implication 1: @Sycorax concern is not an issue. The starting model is a diagonal precision matrix (invertible). Iteratively implemented pairwise covariance constraints to the maximum entropy distribution leave the precision matrix invertible. $\endgroup$
    – krkeane
    Commented Jan 10, 2023 at 13:52
  • 1
    $\begingroup$ @krkeane It seems that you have some expertise in this topic. Perhaps you'd like to ask a question about the misunderstandings surrounding the Dempster 1972 article and write an answer to your question dispels these misunderstandings? This is a Q&A site, so the Q&A site features work best. Moreover, comment boxes are rather constrained. $\endgroup$
    – Sycorax ♦
    Commented Jan 10, 2023 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.