# Asymptotic normality of a quadratic form

Let $\mathbf{x}$ be a random vector drawn from $P$. Consider a sample $\{ \mathbf{x}_i \}_{i=1}^n \stackrel{i.i.d.}{\sim} P$. Define $\bar{\mathbf{x}}_n := \frac{1}{n} \sum_{i=1}^n \mathbf{x}_i$, and $\hat{C} := \frac{1}{n} \sum_{i=1}^n (\mathbf{x}_i - \bar{\mathbf{x}}_n) (\mathbf{x}_i - \bar{\mathbf{x}}_n)^\top$. Let $\boldsymbol{\mu} := \mathbb{E}_{\mathbf{x}\sim P}[\mathbf{x}]$ and $C:=\mathrm{cov}_{\mathbf{x} \sim P}[\mathbf{x}, \mathbf{x}]$.

By the central limit theorem, assume that

$$\sqrt{n} \big( \bar{\mathbf{x}}_n - \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(\boldsymbol{0}, C),$$

where $C$ is a full rank covariance matrix.

Question: How do I prove (or disprove) that

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top (\hat{C} + \gamma_n I)^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top C^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2),$$

for some $v>0$, and for some $\gamma_n \ge 0$ such that $\lim_{n\to \infty} \gamma_n =0$? This looks simple. But I could not figure it out exactly how to show this. This is not a homework question.

My understanding is that the delta method would allow us to easily conclude

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top C^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top C^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2),$$

or

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top (\hat{C} + \gamma_n I)^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top (\hat{C} + \gamma_n I)^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2).$$

These are a bit different from what I want. Notice the covariance matrices in the two terms. I feel that I miss something very trivial here. Alternatively, if it makes things simpler, we can also ignore $\gamma_n$ i.e., set $\gamma_n =0$ and assume that $\hat{C}$ is invertible. Thanks.

• We need to know something about how $\gamma_n$ goes to 0. Is it a sequence of constants? I think that you first have to show $\bar{x}_n^T \gamma_n I \bar{x}_n \rightarrow_p 0$ which I think is a result of Slutsky's. Then I would write $\hat{C}$ as $C + \text{bias}(\hat{C})$. $\bar{x}_n^T C \bar{x}_n$ has a limiting distribution that can be found with $\delta$ method. Lastly you can try to show that $\bar{x}_n^T \text{bias}(\hat{C}) \bar{x}_n$ goes to 0 in probability. Although I'm not sure if that holds... Apr 10 '18 at 13:51
• $\gamma_n$ is a sequence of constants (not random). The sequence can be set to anything that makes the convergence work (if such a sequence exists). I think $\bar{x}_n^\top I \bar{x}_n \stackrel{p}{\to} 0$ is true. I did not follow why we first need this. But let me think about it and the rest more. :)
– wij
Apr 10 '18 at 14:02
• I failed to mention: your hesitation to directly apply the $\delta$-method and call it done is well warranted. I think you can write this out carefully. Useful theorems for these kinds of proofs are Slutsky's, the Mann-Wald Continuous Mapping Theorem, and the Cramer-Wold theorem. Apr 10 '18 at 14:07
• I agree that the results you mention might be useful. I still do not see how though. Actually I also start to think that the asymptotic distribution may not be a normal distribution.
– wij
Apr 12 '18 at 13:18
• It seems like this is more complicated that it seems. The arXiv paper here describes what happens in high-dimensions. I can't find a fixed dimension analog, but they do have a finitie-dimensional argument in Section 3. Apr 16 '18 at 15:45

There is some difficulty when using Delta method. It's more convenient to derive it by hand.

By law of large number, $\hat{C}\xrightarrow{P} C$. Hence $\hat{C} +\gamma_n I\xrightarrow{P} C$. Apply Slutsky's theorem, we have $$\sqrt{n}(\hat{C} +\gamma_n I)^{-1/2}(\bar{X}-\mu)\xrightarrow{d}\mathcal{N} (0,C^{-1}).$$ By continuous mapping theorem, we have $${n}(\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{d}\sum_{i=1}^p \lambda_i^{-1}(C)\chi^2_1.$$ Hence $$\sqrt{n}(\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{P}0.$$ By Slutsky's theorem, we have $$\sqrt{n}\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{d}\mathcal{N} (0,\mu^T C^{-2}\mu).$$ Combining the above two equality yields \begin{align} &\sqrt{n}\big(\bar{X}^T (\hat{C} +\gamma_n I)^{-1}\bar{X}-\mu^T (\hat{C} +\gamma_n I)^{-1}\mu\big) \\ = &\sqrt{n}\Big((\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)-2\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\Big) \\ =&-2\sqrt{n}\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)+o_P(1) \\ \xrightarrow{d}&\mathcal{N} (0,4\mu^T C^{-2}\mu). \end{align} The remaining task is to deal with $$\sqrt{n} \Big( \mu^T (\hat{C} +\gamma_n I)^{-1}\mu-\mu^T (C)^{-1}\mu \Big).$$ Unfortunately, this term dose NOT converges to $0$. The behavior become complicated and depends on the third and fourth moments.

To be simple, below we assume $X_i$ are normal distributed and $\gamma_n=o(n^{-1/2})$. It's a standard result that $$\sqrt{n}(\hat{C}-C)\xrightarrow{d}C^{1/2} W C^{1/2},$$ where $W$ is a symmetric random matrix with diagonal elements as $\mathcal{N}(0,2)$ and off diagonal elements as $\mathcal{N}(0,1)$. Thus, $$\sqrt{n}(\hat{C}+\gamma_n I-C)\xrightarrow{d}C^{1/2} W C^{1/2},$$ By matrix taylor expantion $(I+A)^{-1}\sim I-A+A^2$, we have \begin{align} &\sqrt{n}\Big((\hat{C} +\gamma_n I)^{-1}- C^{-1}\Big)= \sqrt{n}C^{-1/2}\Big(\big(C^{-1/2}(\hat{C} +\gamma_n I)C^{-1/2}\big)^{-1}-I\Big)C^{-1/2}\\ =&\sqrt{n}C^{-1}\Big(\hat{C} +\gamma_n I-C\Big)C^{-1}+O_P(n^{-1/2}) \xrightarrow{d}C^{-1/2} W C^{-1/2}. \end{align} Thus, $$\sqrt{n} \Big( \mu^T (\hat{C} +\gamma_n I)^{-1}\mu-\mu^T (C)^{-1}\mu \Big)\xrightarrow{d}\mu^T C^{-1/2} W C^{-1/2}\mu \sim N(0,(\mu^T C^{-1}\mu)^2).$$

Thus, \begin{align} \sqrt{n}\big(\bar{X}^T (\hat{C} +\gamma_n I)^{-1}\bar{X}-\mu^T C^{-1}\mu\big) \xrightarrow{d}\mathcal{N} (0,4\mu^T C^{-2}\mu+(\mu^T C^{-1}\mu)^2). \end{align}

• Thanks for your answer. It is exactly that term that does not converge to 0 that makes the whole thing difficult. Unfortunately I cannot assume that $X_i$ is normally distributed. But I still appreciate the answer. If you could comment on how it depends on third and fourths moments (perhaps with references), that would be helpful. Also I cannot explain at the moment. But I feel that it $gamma_n$ has to decay slower than $o(n^{-1/2})$. I have to think about the reason more carefully.
– wij
Apr 17 '18 at 11:37
• I forgot to add that in my case $X_i$ can be assumed to live in a compact set (if necessary). This might help with the moment conditions.
– wij
Apr 17 '18 at 12:53