Asymptotic normality of a quadratic form

Question

Let $\mathbf{x}$ be a random vector drawn from $P$. Consider a sample $\{ \mathbf{x}_i \}_{i=1}^n \stackrel{i.i.d.}{\sim} P$. Define $\bar{\mathbf{x}}_n := \frac{1}{n} \sum_{i=1}^n \mathbf{x}_i$, and $\hat{C} := \frac{1}{n} \sum_{i=1}^n (\mathbf{x}_i - \bar{\mathbf{x}}_n) (\mathbf{x}_i - \bar{\mathbf{x}}_n)^\top$. Let $\boldsymbol{\mu} := \mathbb{E}_{\mathbf{x}\sim P}[\mathbf{x}]$ and $C:=\mathrm{cov}_{\mathbf{x} \sim P}[\mathbf{x}, \mathbf{x}]$.

By the central limit theorem, assume that

$$ \sqrt{n} \big( \bar{\mathbf{x}}_n - \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(\boldsymbol{0}, C), $$

where $C$ is a full rank covariance matrix.

Question: How do I prove (or disprove) that

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top (\hat{C} + \gamma_n I)^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top C^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2),$$

for some $v>0$, and for some $\gamma_n \ge 0$ such that $\lim_{n\to \infty} \gamma_n =0$? This looks simple. But I could not figure it out exactly how to show this. This is not a homework question.

My understanding is that the delta method would allow us to easily conclude

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top C^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top C^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2),$$

or

$$\sqrt{n} \big( \bar{\mathbf{x}}_n^\top (\hat{C} + \gamma_n I)^{-1} \bar{\mathbf{x}}_n - \boldsymbol{\mu}^\top (\hat{C} + \gamma_n I)^{-1} \boldsymbol{\mu} \big) \stackrel{d}{\to} \mathcal{N}(0, v^2).$$

These are a bit different from what I want. Notice the covariance matrices in the two terms. I feel that I miss something very trivial here. Alternatively, if it makes things simpler, we can also ignore $\gamma_n$ i.e., set $\gamma_n =0$ and assume that $\hat{C}$ is invertible. Thanks.

Answer 1

There is some difficulty when using Delta method. It's more convenient to derive it by hand.

By law of large number, $\hat{C}\xrightarrow{P} C$. Hence $\hat{C} +\gamma_n I\xrightarrow{P} C$. Apply Slutsky's theorem, we have $$\sqrt{n}(\hat{C} +\gamma_n I)^{-1/2}(\bar{X}-\mu)\xrightarrow{d}\mathcal{N} (0,C^{-1}).$$ By continuous mapping theorem, we have $$ {n}(\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{d}\sum_{i=1}^p \lambda_i^{-1}(C)\chi^2_1. $$ Hence $$ \sqrt{n}(\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{P}0. $$ By Slutsky's theorem, we have $$ \sqrt{n}\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\xrightarrow{d}\mathcal{N} (0,\mu^T C^{-2}\mu). $$ Combining the above two equality yields \begin{align} &\sqrt{n}\big(\bar{X}^T (\hat{C} +\gamma_n I)^{-1}\bar{X}-\mu^T (\hat{C} +\gamma_n I)^{-1}\mu\big) \\ = &\sqrt{n}\Big((\bar{X}-\mu)^T (\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)-2\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)\Big) \\ =&-2\sqrt{n}\mu^T(\hat{C} +\gamma_n I)^{-1}(\bar{X}-\mu)+o_P(1) \\ \xrightarrow{d}&\mathcal{N} (0,4\mu^T C^{-2}\mu). \end{align} The remaining task is to deal with $$ \sqrt{n} \Big( \mu^T (\hat{C} +\gamma_n I)^{-1}\mu-\mu^T (C)^{-1}\mu \Big). $$ Unfortunately, this term dose NOT converges to $0$. The behavior become complicated and depends on the third and fourth moments.

To be simple, below we assume $X_i$ are normal distributed and $\gamma_n=o(n^{-1/2})$. It's a standard result that $$ \sqrt{n}(\hat{C}-C)\xrightarrow{d}C^{1/2} W C^{1/2}, $$ where $W$ is a symmetric random matrix with diagonal elements as $\mathcal{N}(0,2)$ and off diagonal elements as $\mathcal{N}(0,1)$. Thus, $$ \sqrt{n}(\hat{C}+\gamma_n I-C)\xrightarrow{d}C^{1/2} W C^{1/2}, $$ By matrix taylor expantion $(I+A)^{-1}\sim I-A+A^2$, we have \begin{align} &\sqrt{n}\Big((\hat{C} +\gamma_n I)^{-1}- C^{-1}\Big)= \sqrt{n}C^{-1/2}\Big(\big(C^{-1/2}(\hat{C} +\gamma_n I)C^{-1/2}\big)^{-1}-I\Big)C^{-1/2}\\ =&\sqrt{n}C^{-1}\Big(\hat{C} +\gamma_n I-C\Big)C^{-1}+O_P(n^{-1/2}) \xrightarrow{d}C^{-1/2} W C^{-1/2}. \end{align} Thus, $$ \sqrt{n} \Big( \mu^T (\hat{C} +\gamma_n I)^{-1}\mu-\mu^T (C)^{-1}\mu \Big)\xrightarrow{d}\mu^T C^{-1/2} W C^{-1/2}\mu \sim N(0,(\mu^T C^{-1}\mu)^2).$$

Thus, \begin{align} \sqrt{n}\big(\bar{X}^T (\hat{C} +\gamma_n I)^{-1}\bar{X}-\mu^T C^{-1}\mu\big) \xrightarrow{d}\mathcal{N} (0,4\mu^T C^{-2}\mu+(\mu^T C^{-1}\mu)^2). \end{align}