12
$\begingroup$

Let $\mathbf{x}$ be a random vector drawn from $P$. Consider a sample $\{ \mathbf{x}_i \}_{i=1}^n \stackrel{i.i.d.}{\sim} P$. Define $\bar{\mathbf{x}}_n := \frac{1}{n} \sum_{i=1}^n \mathbf{x}_i$, and $\hat{C} := \frac{1}{n} \sum_{i=1}^n (\mathbf{x}_i - \bar{\mathbf{x}}_n) (\mathbf{x}_i - \bar{\mathbf{x}}_n)^\top$. Let $\boldsymbol{\mu} := \mathbb{E}_{\mathbf{x}\sim P}[\mathbf{x}]$ and $C:=\mathrm{cov}_{\mathbf{x} \sim P}[\mathbf{x}, \mathbf{x}]$.

By the central limit theorem, assume that

$$ \sqrt{n} \big( \bar{\mathbf{x}}_n - \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(\boldsymbol{0}, C), $$

where $C$ is a full rank covariance matrix.

Question: How do I prove (or disprove) that

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top (\hat{C} + \gamma_n I)^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top C^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2),$$

for some $v>0$, and for some $\gamma_n \ge 0$ such that $\lim_{n\to \infty} \gamma_n =0$? This looks simple. But I could not figure it out exactly how to show this. This is not a homework question.

My understanding is that the delta method would allow us to easily conclude

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top C^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top C^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2),$$

or

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top (\hat{C} + \gamma_n I)^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top (\hat{C} + \gamma_n I)^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2).$$

These are a bit different from what I want. Notice the covariance matrices in the two terms. I feel that I miss something very trivial here. Alternatively, if it makes things simpler, we can also ignore $\gamma_n$ i.e., set $\gamma_n =0$ and assume that $\hat{C}$ is invertible. Thanks.

$\endgroup$
6
  • 2
    $\begingroup$ We need to know something about how $\gamma_n$ goes to 0. Is it a sequence of constants? I think that you first have to show $\bar{x}_n^T \gamma_n I \bar{x}_n \rightarrow_p 0$ which I think is a result of Slutsky's. Then I would write $\hat{C}$ as $C + \text{bias}(\hat{C})$. $\bar{x}_n^T C \bar{x}_n$ has a limiting distribution that can be found with $\delta$ method. Lastly you can try to show that $\bar{x}_n^T \text{bias}(\hat{C}) \bar{x}_n$ goes to 0 in probability. Although I'm not sure if that holds... $\endgroup$
    – AdamO
    Apr 10, 2018 at 13:51
  • $\begingroup$ $\gamma_n$ is a sequence of constants (not random). The sequence can be set to anything that makes the convergence work (if such a sequence exists). I think $\bar{x}_n^\top I \bar{x}_n \stackrel{p}{\to} 0$ is true. I did not follow why we first need this. But let me think about it and the rest more. :) $\endgroup$
    – wij
    Apr 10, 2018 at 14:02
  • 2
    $\begingroup$ I failed to mention: your hesitation to directly apply the $\delta$-method and call it done is well warranted. I think you can write this out carefully. Useful theorems for these kinds of proofs are Slutsky's, the Mann-Wald Continuous Mapping Theorem, and the Cramer-Wold theorem. $\endgroup$
    – AdamO
    Apr 10, 2018 at 14:07
  • $\begingroup$ I agree that the results you mention might be useful. I still do not see how though. Actually I also start to think that the asymptotic distribution may not be a normal distribution. $\endgroup$
    – wij
    Apr 12, 2018 at 13:18
  • $\begingroup$ It seems like this is more complicated that it seems. The arXiv paper here describes what happens in high-dimensions. I can't find a fixed dimension analog, but they do have a finitie-dimensional argument in Section 3. $\endgroup$ Apr 16, 2018 at 15:45

1 Answer 1

2
$\begingroup$

There is some difficulty when using Delta method. It's more convenient to derive it by hand.

By law of large number, $\hat{C}\xrightarrow{P} C$. Hence $\hat{C} +\gamma_n I\xrightarrow{P} C$. Apply Slutsky's theorem, we have $$\sqrt{n}(\hat{C} +\gamma_n I)^{-1/2}(\bar{X}-\mu)\xrightarrow{d}\mathcal{N} (0,C^{-1}).$$ By continuous mapping theorem, we have $$ {n}(\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{d}\sum_{i=1}^p \lambda_i^{-1}(C)\chi^2_1. $$ Hence $$ \sqrt{n}(\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{P}0. $$ By Slutsky's theorem, we have $$ \sqrt{n}\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{d}\mathcal{N} (0,\mu^T C^{-2}\mu). $$ Combining the above two equality yields \begin{align} &\sqrt{n}\big(\bar{X}^T (\hat{C} +\gamma_n I)^{-1}\bar{X}-\mu^T (\hat{C} +\gamma_n I)^{-1}\mu\big) \\ = &\sqrt{n}\Big((\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)-2\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\Big) \\ =&-2\sqrt{n}\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)+o_P(1) \\ \xrightarrow{d}&\mathcal{N} (0,4\mu^T C^{-2}\mu). \end{align} The remaining task is to deal with $$ \sqrt{n} \Big( \mu^T (\hat{C} +\gamma_n I)^{-1}\mu-\mu^T (C)^{-1}\mu \Big). $$ Unfortunately, this term dose NOT converges to $0$. The behavior become complicated and depends on the third and fourth moments.

To be simple, below we assume $X_i$ are normal distributed and $\gamma_n=o(n^{-1/2})$. It's a standard result that $$ \sqrt{n}(\hat{C}-C)\xrightarrow{d}C^{1/2} W C^{1/2}, $$ where $W$ is a symmetric random matrix with diagonal elements as $\mathcal{N}(0,2)$ and off diagonal elements as $\mathcal{N}(0,1)$. Thus, $$ \sqrt{n}(\hat{C}+\gamma_n I-C)\xrightarrow{d}C^{1/2} W C^{1/2}, $$ By matrix taylor expantion $(I+A)^{-1}\sim I-A+A^2$, we have \begin{align} &\sqrt{n}\Big((\hat{C} +\gamma_n I)^{-1}- C^{-1}\Big)= \sqrt{n}C^{-1/2}\Big(\big(C^{-1/2}(\hat{C} +\gamma_n I)C^{-1/2}\big)^{-1}-I\Big)C^{-1/2}\\ =&\sqrt{n}C^{-1}\Big(\hat{C} +\gamma_n I-C\Big)C^{-1}+O_P(n^{-1/2}) \xrightarrow{d}C^{-1/2} W C^{-1/2}. \end{align} Thus, $$ \sqrt{n} \Big( \mu^T (\hat{C} +\gamma_n I)^{-1}\mu-\mu^T (C)^{-1}\mu \Big)\xrightarrow{d}\mu^T C^{-1/2} W C^{-1/2}\mu \sim N(0,(\mu^T C^{-1}\mu)^2).$$

Thus, \begin{align} \sqrt{n}\big(\bar{X}^T (\hat{C} +\gamma_n I)^{-1}\bar{X}-\mu^T C^{-1}\mu\big) \xrightarrow{d}\mathcal{N} (0,4\mu^T C^{-2}\mu+(\mu^T C^{-1}\mu)^2). \end{align}

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your answer. It is exactly that term that does not converge to 0 that makes the whole thing difficult. Unfortunately I cannot assume that $X_i$ is normally distributed. But I still appreciate the answer. If you could comment on how it depends on third and fourths moments (perhaps with references), that would be helpful. Also I cannot explain at the moment. But I feel that it $gamma_n$ has to decay slower than $o(n^{-1/2})$. I have to think about the reason more carefully. $\endgroup$
    – wij
    Apr 17, 2018 at 11:37
  • $\begingroup$ I forgot to add that in my case $X_i$ can be assumed to live in a compact set (if necessary). This might help with the moment conditions. $\endgroup$
    – wij
    Apr 17, 2018 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.