3
$\begingroup$

Assume we have $n$ iid random vectors $y_1, y_2, …, y_n$, normally distributed with zero mean and unknown covariance matrix $M$. Each vector is of size $p$. I know a lot of methods that provide a sparse estimation of the true unknown covariance matrix.

My question: Consider the case when $p\gg n$. Is it possible to have an estimate covariance matrix which is better than the true one? I mean by better that I can use the new estimate covariance instead of the true one (assumed to be known now) in specific applications such as hyperspectral image detection, classification, etc.

For example: Take the article of Ledoit and Wolf entitled "A well-conditioned estimator for large-dimensional covariance matrices". They developed a new estimator which is the weighted average between the sample covariance and the identity matrix. They mentioned in the article that the new estimator is more accurate than either of them (page 2). So if we consider that the true covariance matrix, which is unknown, is indeed the identity (just create this hypothesis), we can expect that the new estimator of Ledoit and Wolf is better. Is it not logical what I am assuming?

Any help will be very appreciated!

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ Why would you want to use an estimate if you know the true value? Also, I am trying to understand what $n$ and $p$ are in this context, wondering if it is not more usual to use an inverted notation ($n$ in place of $p$ and vice versa). Do you have $n$ realizations of a $p$-variate random vector, or do you have $p$ realizations of an $n$-variate random vector? I think the former would be the standard notation while the latter would be inverted. And so when you say $p \gg n$, do you mean the dimension of the random variable is larger than the number of realizations, or vice versa? $\endgroup$ – Richard Hardy Sep 20 '15 at 11:34
  • $\begingroup$ You can say that $n$ corresponds to the secondary data used to estimate the unknown covariance matrix and $p$ the matrix dimension. So the size of the covariance matrix increases quadratically with p. $\endgroup$ – Christina Sep 20 '15 at 11:37
  • 1
    $\begingroup$ So $n$ is the sample size (the number of realizations)? If you add a few more observations, it will be $n$ that will increase? $\endgroup$ – Richard Hardy Sep 20 '15 at 11:39
  • $\begingroup$ Yes. So when $p>n$, the sample covariance matrix becomes not invertible (singular). $\endgroup$ – Christina Sep 20 '15 at 11:50
  • 1
    $\begingroup$ OK, I got it. I wonder about the idea why an estimate will be "better" than the true value. Could you provide some intuition (maybe by expanding your post instead of posting a comment)? $\endgroup$ – Richard Hardy Sep 20 '15 at 11:57
4
$\begingroup$

What you are describing sounds to me as ridge-regression or Tikhonov regularization. You add a ridge to the diagonal, i.e. a scaled identity matrix.

The problem is that if you have more variables than observations, i.e. $p>>n$, you cannot estimate the parameters in some models, e.g. a linear model. If you have a model: $$ \mathbf{y} = \mathbf{X}\beta + \epsilon $$ Where $\mathbf{y}$ is $n\times 1$ and $\mathbf{X}$ is $n\times p$. Now the estimate of $\beta$ is of the form: $$ \hat{\beta} = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y} $$

Note that the matrix $\mathbf{X}^T \mathbf{X}$ will be rank deficient if $p>>n$, (here the matrix $\sigma^2(\mathbf{X}^T \mathbf{X})^{-1}$ is the covariance matrix of the parameters $\beta$). Thus we need to add some form of regularization to be able to get a solution, because that requires us to invert this matrix. One such type is the one you mention.

So this is better, because we cannot get any estimate unless we throw away some of the variables or add some form of regularization.

EDIT: To address your question of if this estimate presented in the paper is better than the true covariance matrix you should read over the conlusions in the paper:

In this paper, we have discussed the estimation of large-dimensional covariance matrices where the number of (iid) variables is not small compared to the sample size. It is well-known that in such situations the usual estimator, the sample covariance matrix, is ill-conditioned and may not even be invertible. The approach suggested is to shrink the sample covariance matrix towards the identity matrix, which means to consider a convex linear combination of these two matrices. The practical problem is to determine the shrinkage intensity, that is, the amount of shrinkage of the sample covariance matrix towards the identity matrix. To solve this problem, we considered a general asymptotics framework where the number of variables is allowed to tend to infinity with the sample size. It was seen that under mild conditions the optimal shrinkage intensity then tends to a limiting constant; here, optimality is meant with respect to a quadratic loss function based on the Frobenius norm. It was shown that the asymptotically optimal shrinkage intensity can be estimated consistently, which leads to a feasible estimator. Both the asymptotic results and the extensive Monte-Carlo simulations presented in this paper indicate that the suggested shrinkage estimator can serve as an all-purpose alternative to the sample covariance matrix. It has smaller risk and is better- conditioned. This is especially true when the dimension of the covariance matrix is large compared to the sample size

Thus, the estimate they provide is being compared to the sample covariance estimate. Not the true underlying covariance matrix.

EDIT2: The way the authors describe this as better, (on page 3 in the manuscript), refers to the condition number of the matrix. That means that their estimate is more numerically stable. This is usually the case when you perform any kind of regularization, since you are reducing the effective number of parameters that you are estimating.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Nice answer. I just wonder what your take on Is it possible to have an estimate covariance matrix which is better than the true one? is. $\endgroup$ – Richard Hardy Sep 20 '15 at 15:49
  • $\begingroup$ I do not think that is possible. What is essentially being done in this paper OP references, is that bias is being traded in for variance. $\endgroup$ – Gumeo Sep 20 '15 at 16:03
  • $\begingroup$ OK. I am not sure I entirely understood the original question. Using something else (some estimator) when the true value is known and expecting better results (in whatever sense) is a confusing notion in itself. $\endgroup$ – Richard Hardy Sep 20 '15 at 16:08
  • $\begingroup$ I think that OP might be referencing the traditional estimate of the covariance matrix, but not the true underlying value. I was also confused by this question. $\endgroup$ – Gumeo Sep 20 '15 at 16:10
  • 1
    $\begingroup$ I can do better than Ledoit and Wolf - give me some data, and i can return an estimated covariance with the lowest possible condition number. My estimate will have a lower condition number than the true covariance, except when the true covariance is a scalar multiple of the identity matrix. Have you guessed by now that my estimate is (a scalar multiple of) the identity matrix? So this is what Ledoit and Wolf do, just not to this extreme. They blend in a scalar multiple of the identity matrix with the sample covariance, and provide a method for determining the blending (weights). That's all. $\endgroup$ – Mark L. Stone Sep 20 '15 at 18:13

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.