# Suppose $E[( \theta-\hat{\theta}_n)^2] = O(n^{-1/2})$. Show that $\frac{1}{n}\sum_{i=1}^n (\theta-\hat{\theta}_n)^2 = O_p(n^{-1/2}).$

Assume that $$E[( \theta-\hat{\theta}_n)^2] = O(n^{-1/2})$$. How can I show that $$\frac{1}{n}\sum_{i=1}^n (\theta-\hat{\theta}_n)^2 = O_p(n^{-1/2})?$$

What I'm trying to ask is: if the expected value of some quantity is $$O(n^{-1/2})$$, does that imply the sample mean of that quantity is also $$O_p(n^{-1/2})$$?

• Are you sure you have the subscript notation right? You’re using the same stuff for random sequences and nonrandom sequences Commented Mar 19, 2022 at 17:34
• @Taylor Could you elaborate? Are you talking about the subscript $n$ for $\hat{\theta}$? Commented Mar 19, 2022 at 17:55
• Sorry if my notation is confusing. What I'm trying to ask is: if the expected value of some quantity is $O_p(n^{-1/2})$, does that imply the sample mean of that quantity is also $O_p(n^{-1/2})$? Commented Mar 19, 2022 at 18:03
• It's not confusing, it's just incorrect. $O_p$ is not the same as $O$ @Adrian Commented Mar 20, 2022 at 2:58
• @Taylor can you point me to a resource that explains the difference between the two notation? Commented Mar 20, 2022 at 3:09

You want to show that for any $$\varepsilon>0$$, there exist two positive reals $$M$$ and $$N$$ such that $$\mathbb P\left(\frac{1}{\sqrt n}\sum_{i=1}^n (\theta-\hat{\theta}_n)^2 > M\right)\le\varepsilon\ \ \forall n\ge N$$ Let $$\varepsilon>0$$. For any $$n\ge1$$ and $$M>0$$ you have, by Markov inequality, that \begin{align}\mathbb P\left(\frac{1}{\sqrt n}\sum_{i=1}^n (\theta-\hat{\theta}_n)^2 > M\right)&\le\frac 1 M\mathbb E\left[\frac{1}{\sqrt n}\sum_{i=1}^n (\theta-\hat{\theta}_n)^2 \right]\\ &\le\frac{1}{M\sqrt n}\mathbb E\left[\sum_{i=1}^n (\theta-\hat{\theta}_n)^2 \right]\\ &\le\frac{\sqrt n}{M}\mathbb E\left[(\theta-\hat{\theta}_n)^2 \right]\\\end{align} But, because you assume that $$E[( \theta-\hat{\theta}_n)^2] = O(n^{-1/2})$$ (it's a deterministic sequence so I assume the big O is the usual one, otherwise it makes no sense), you know that for any positive $$M'$$, there exists $$N'$$ such that, for all $$n\ge N'$$ : $$\sqrt nE[( \theta-\hat{\theta}_n)^2] \le M'$$ Now take $$M'=M\varepsilon$$ and you can conclude that the desired inequality holds for all $$n\ge N'$$.

[Note that I assumed that the $$\theta$$ in the sum were supposed to be $$n$$ i.i.d. realizations of $$\theta$$ called $$\theta_i$$.]

• Should it be $P\left(\sqrt{n}\sum_{i=1}^n (\theta - \hat{\theta}_n)^2 > M\right) \leq \epsilon$ in the first line? Because I want to show that $\sum_{i=1}^n (\theta - \hat{\theta}_n)^2$ is $O(n^{-1/2})$. Commented Mar 20, 2022 at 1:39
• If so, then I have $P\left(\sqrt{n} \sum_{i=1}^n (\theta - \hat{\theta}_n)^2 > M\right) \leq \frac{n}{M} \sqrt{n}E[(\theta - \hat{\theta}_n)^2]$. Since $E[(\theta - \hat{\theta})^2] = O(n^{-1/2})$, then $\sqrt{n}E[(\theta - \hat{\theta}_n)^2] \leq M'$. Therefore, should I take $M' = \frac{M\epsilon}{n}?$ Is it OK for $M'$ to depend on $n$? Commented Mar 20, 2022 at 1:59
• No, you are trying to prove that $$U_n =\color{red}{\frac{1}{n}}\sum_{i=1}^n (\theta-\hat{\theta}_n)^2$$ Is $O_p(n^{-1/2})$, so when you apply the definition you end up with the expression I've written. Regarding your second question, $M'$ can not depend on $n$ as it is supposed to be a constant (again, look at the definition from Wikipédia). Commented Mar 21, 2022 at 0:29

### Big-O in terms of the quantile function

We can interpret the $$O_p$$ in terms of the quantile functions.

If

$$X_n = {O}_p(a_n) \quad \text{for }n \to \infty$$

then in terms of the quantile functions for $$X_n$$

$$\forall p \in (0,1): \lbrace Q_{X_n}(p) = {O}(a_n) \quad \text{for }n \to \infty \rbrace$$

See this also in the question: What is difference between $\hat{X}_n \overset{p}{\to} \bar{x}$ and $(\hat{X}_n - \bar{x}) = o_p(1)$?

### Expectation in terms of the quantile function

If the quantile function is continuous, then we can express the expectation value as an integral over the quantile function

$$E[X_n] = \int_0^1 Q_{X_n}(p) \,\text{d}p$$

And for non-negative variables $$X_n$$ we have for every $$p_c<1$$

$$E[X_n] \geq \int_{p_c}^1 Q_{X_n}(p) \,\text{d}p \geq \int_{p_c}^1 Q_{X_n}(p_c) \,\text{d}p = (1-p_c) Q_{X_n}(p_c)$$

From which it follows that $$Q_{X_n}(p_c) = O\left(E[X_n]\right)$$ and the quantiles $$p_c<1$$ have at least the same order of convergence as the expectation value.

### Edge case

Say the quantiles function is $$Q_{X_n}(p) = \log(n) n^{p/2-1}$$ then $$E[X_n] = 2n^{-0.5} + 2n^{-n} = O\left(n^{-0.5}\right)$$

The quantile functions have convergence $$O\left(\log(n)n^{-a}\right)$$ with $$a = 0.5 \cdot (2-p)$$ For $$p =1$$ this power is $$0.5$$ and the convergence bound is $$O\left(\log(n)n^{-0.5}\right)$$, which means less fast convergence. However, for any $$p<1$$ the power is $$a>0.5$$ and the convergence is faster than the convergence of the expectation. We can only make examples where the quantiles $$p=1$$ converges less fast.

• The argument here is basically the same as the use of the Markov inequality. Commented Aug 2 at 7:43