3
$\begingroup$

I've been reading this code (based on this R package) and I found that the number of non-zero eigenvalues of the estimated covariance is roughly equal to $x_i^T \hat{\Sigma}^{-1} x_i$. I want to know how to arrive at this result.

This arises in the context of Maximum Likelihood estimation of generalized ARMA coefficients. I've made a few tests with data generated from a multivariate normal distribution with random covariance matrices and the results are consistent.

It seems I'm lacking of a little bit of linear algebra

Some background:

  • $x_i$ is a real column vector with dimension $d$ (one observation)
  • $X = [x_1, x_2, \ldots, x_n]$ with shape $d\times n$ (all the observations)

I want to prove that:
$$\sum_{i=0}^{i=n} x_i^T \hat{\Sigma}^{-1} x_i = n\operatorname{len}(s)$$ being $\operatorname{len}(s)$ the number of non-zero singular values* of $\hat{\Sigma}$, that is defined as
$$\hat{\Sigma} = \frac{1}{n} \sum_{j=1}^{j=n} x_j x_j^T$$

If necessary, mean can be considered $0$

*Not necessarily mathematical $0$, it can also mean "not too small values" .
Actually, "non-zero" means "non-negligible" depending on a threshold defined as the largest singular value times the square root of the machine epsilon.
In Python: s[0] * np.sqrt(np.finfo(np.float).eps) being s the singular values in descending order (see the code)

$\endgroup$
  • $\begingroup$ Are you sure $\Sigma$ and $x_i$ are related? If so, then you are looking at fourth-order statistic. In general, we replace $\Sigma$ with $\hat{\Sigma}$ and consider it as a constant matrix, and then apply quadratic forms of random variables. $\endgroup$ – Maxtron Nov 20 '18 at 2:03
  • $\begingroup$ As often happens here, you're using the word "sample" incorrectly. The vector $x_i$, with $d$ scalar entries, is not one sample; it is one observation within a sample. The whole sequence $x_1,\ldots,x_n,$ with $n\cdot d$ scalar entries, is the whole sample. Just one whole sample. $\endgroup$ – Michael Hardy Nov 20 '18 at 3:58
  • $\begingroup$ In order to make any sense of this question you need to find a reasonable definition of $\Sigma^{-1}$ for singular $\Sigma.$ In exploring that issue you should quickly understand what's going on. Also, if you consider for a moment the case $d=1,$ you will see your result cannot possibly be generally true: it looks like you are assuming the rows of $x$ have been centered. $\endgroup$ – whuber Nov 20 '18 at 13:53
  • $\begingroup$ Thanks for the helpful comments. @Maxtron, indeed, $\Sigma$ as defined above is $\hat{\Sigma}$. Sorry for the notation, fixed now. $\endgroup$ – Franco Marchesoni Nov 20 '18 at 15:17
  • $\begingroup$ @MichaelHardy , thanks for the correction. Probably this kind of mistakes are due to translations from Spanish (as in my case). Edited too $\endgroup$ – Franco Marchesoni Nov 20 '18 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.