Convergence in probability in high-dimensional settings

Question

I am trying to prove the following.

$$ (\hat{\theta} - \theta)^T c \rightarrow 0 \;\;\; \text{ (in probability)}$$

where $\theta \in R^p$, $c$ is a vector of constants such that $\sup_i (|c_i|) < \infty$ ($c_i$ denoting the $i$-th element of $c$) and $p \rightarrow \infty$ with the sample size $n$. It is know that

$$ \sqrt{n} (\hat{\theta}_i - \theta_i) \rightarrow N(0, \sigma_i^2) \;\;\; \text{ (in distribution)}$$

where $\sigma_i^2 < \infty$. Here is what I did:

$$(\hat{\theta} - \theta)^T c \leq \sum_{i=1}^{p} |(\hat{\theta}_i - \theta_i) c_i | \leq p \max_i (| (\hat{\theta}_i - \theta_i) c_i |) = \mathcal{O}_p(p/\sqrt{n}).$$

Therefore, $(\hat{\theta} - \theta)^T c \rightarrow 0$ if $p/\sqrt{n} \rightarrow 0$. Is there a smarter way of doing this? In particular, would the condition $p/n \rightarrow 0$ be sufficient?

Answer 1

Suppose the $\hat\theta_i$ are perfectly correlated and $c_i$ are identical. Then your bound is sharp, so $p/\sqrt{n}\to 0$ is necessary

You can do worse: convergence in distribution of $\hat\theta_i$ to $\theta_i$ does not guarantee anything about the maximum at finite $n$, so your proof is actually wrong.

Suppose that $\hat\theta$ is iid $N(0,1/n)$ except that one element of $\hat\theta_i$, chosen at uniformly at random, is set to 42. As long as $p\to\infty$ it is still true that $$\sqrt{n}(\hat\theta_i-\theta_i)\stackrel{d}{\to}N(0,1)$$ since the probability of the outlying value is $1/p$ for each $i$ and converges to zero. Now take $c_i=\kappa$ constant: $$(\hat\theta-\theta)c \sim N(0,\kappa^2p/n)+42\kappa$$ and this does not converge to zero no matter how big $n$ is.

Answer 2

I assume that the dependence on $n$ is through $\hat \theta = \hat \theta_n$, and that $\theta$ is fixed.

Let $\varepsilon > 0$ and $\gamma > 0$. Let $F_n$ be the distribution function of $n \|\hat \theta_n - \theta\|^2$, i.e. $F_n(y) = \mathbb P(n \|\hat \theta_n - \theta\|^2 \leq y)$. The stated assumption on convergence in distribution means that $F_n(y) \rightarrow F(y)$, where $F(y)$ is the limiting distribution.

Let $N$ sufficiently large so that for $n \geq N$, $\sup_{y \geq 0} |F_n(y) - F(y)| < \varepsilon/2$ and $F(n \gamma^2) > 1 - \varepsilon/2$.

Now for $n \geq N$, $$\mathbb P(\|\hat \theta_n - \theta\| > \gamma) = \mathbb P(n \| \hat \theta_n - \theta\|^2 > n \gamma^2) \rightarrow 1 - F_n(n \gamma^2) \leq 1 - F(n \gamma^2) + \varepsilon/2 < \varepsilon.$$

This shows that $\hat \theta_n \rightarrow \theta$ in probability and in particular for any $c \in \mathbb R^p$, $(\hat \theta_n - \theta) \cdot c \rightarrow 0$ in probability.