Hard thresholding a covariance matrix

Question

I am new to the concept of thresholding a variance-covariance matrix and am having trouble understanding the exact process. I am following Bickel and Levina (2008) in choosing a hard threshold. What troubles me is their equation number (3) for the threshold operator:

$$ T_{s}(M) = [m_{ij}1(|m_{ij}| \ge s)] $$

My interpretation of that equation is that the thresholding operation applies to the diagonal elements of matrix $M$. This doesn't make much sense to me. In a variance-covariance matrix I am not sure why you would want to set any of the variances equal to zero.

To be explicit, my questions are:

• Does the thresholding operator apply to the diagonal elements?
• If I only apply the thresholding operator to off-diagonal elements will that result in a bad estimate of the variance-covariance matrix?

The context which my problem comes up is I am estimating a probit model with endogenous regressors via generalized method of moments following Wilde (2008). I have a large number of regressors and a number of them are indicator variables. With some specifications of the model the variance-covariance matrix is singular which presents a problem. I am open to any and all solutions but one solution I read about is this thresholding operation.

I want to mention that I am going to bundle the estimation of an endogenous probit model via GMM into an R package. I would really appreciate any help on making it robust and useful to the statistical/econometric community.

Answer 1

In that paper, they restrict the analysis to cases where $T_s(M)$ remains a positive-definite matrix, which implies that $s$ must be smaller than any diagonal element of $M$. However, the condition $s<m_{ii}$ for all $i$ is not enough to guarantee that $T_s(M)$ is positive definite. Since an estimated covariance matrix which is not positive definite is not very useful, then if you are implementing this in software, you might want to report an error in such situations.

One question is whether it is possible to determine whether $T_s(M)$ will be positive definite implicitly without doing the actual computation. The condition given in the paper which guarantees that $T_s(M)$ is positive definite is $$||T_s(M)-M|| < \lambda_{\text{min}}(M), $$ where $||\cdot||$ is the operator norm (w.r.t. $L_2$) and $\lambda_{\text{min}}(M)$ is the smallest eigenvalue of $M$.

They also use the bound $||M||\leq \text{max}_j \sum_i |m_{ij}|$ on symmetric matrices to deduce that if $$\text{max}_j\sum_i |m_{ij}|1(|m_{ij}<s|) <\lambda_\text{min}(M)$$ then $T_s(M)$ must be positive definite, and they analyze a class of matrices where this will be the case.

Of course, going this route requires finding the smallest eigenvalue of $M$, which is harder than just directly determining whether $T_s(M)$ is positive definite, so it may not be helpful in your case, unless maybe you want to try a whole bunch of different threshold values for the same $M$.