# $E[(\beta_n-\beta)^2|\mathbf{X}]=\sigma^2(\mathbf{X}^T\mathbf{X})^{-1}$, what about $E[(\beta_n-\beta)^k|\mathbf{X}]$ for $k=3,4$?

Ultimately my goal with this post is to find the orders of magnitude of $$E[(\beta_n-\beta)^k]$$ for $$k\ge 3$$ as $$n \to \infty$$ so that I can be sure that is safe to drop higher order terms in a Taylor expansion based approximation. As a first step towards $$E[(\beta_n-\beta)^k]$$ I consider $$E[(\beta_n-\beta)^k|\mathbf{X}]$$ in what follows.

My real interest is in the case of multiple regression (including an intercept term) but I think it is easier to focus on a simpler case first. Suppose we have the simple intercept-free linear regression model $$Y_i = \beta X_i + \varepsilon_i, \quad \quad i=1,\dots,n,$$ where $$E[\varepsilon_i|\mathbf{X}] = 0$$, $$\text{Var}[\varepsilon_i^2|\mathbf{X}] = \sigma^2$$, and $$E[\varepsilon_i\varepsilon_j|\mathbf{X}] = 0$$ for $$i\neq j$$.

It is known that $$\hat \beta = \beta + A\varepsilon$$ where $$A = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'.$$ In this simple setting the data matrix $$\mathbf{X}$$ is actually a vector: $$\mathbf{X} = [x_1,\dots,x_n]^T.$$ So $$A\varepsilon$$ is simply a scalar in our simple model.

Now consider evaluating $$E[(\hat \beta - \beta)^3|\mathbf{X}]$$ or $$E[(\hat \beta - \beta)^4|\mathbf{X}]$$ using the same process (see p. 61, Econometric Analysis 8 ed. by Green or p.29, Econometrics by Hayashi) that is used for finding an expression for $$E[(\hat \beta - \beta)^2|\mathbf{X}]$$. That is \begin{align} E[(\hat \beta - \beta)^2|\mathbf{X}] &= E[(A\varepsilon)^2|\mathbf{X}] \\ & = A E[\varepsilon \varepsilon^T|\mathbf{X}]] A^T \\ & = A \sigma^2 A^T \quad \quad \quad \quad \quad \quad \quad \quad \text{(using assumption E[\varepsilon \varepsilon^T|\mathbf{X}]] = \sigma^2 )}\\ & = \sigma^2 (\mathbf{X}^T\mathbf{X})^{-1}. \end{align}

Now lets apply this procedure to the higher moments: \begin{align} E[(\hat \beta - \beta)^3|\mathbf{X}] &= E[(A\varepsilon)^3|\mathbf{X}] \\ & = A E[\varepsilon (A \varepsilon) \varepsilon^T|\mathbf{X}]] A^T \\ \end{align}

and

\begin{align} E[(\hat \beta - \beta)^4|\mathbf{X}] &= E[(A\varepsilon)^4|\mathbf{X}] \\ & = A E[\varepsilon (A \varepsilon)^2 \varepsilon^T|\mathbf{X}]] A^T \\ \end{align}

In both cases we get stuck with $$A$$'s inside the expectation. So is there any way to overcome this and obtain explicit expressions for $$E[(\hat \beta - \beta)^3|\mathbf{X}]$$ and $$E[(\hat \beta - \beta)^4|\mathbf{X}]$$?} This issue is overcome in the attempt below.

Further attempt: We have $$A = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}' = ([x_1, \dots, x_n] [x_1, \dots, x_n]^T)^{-1} [x_1, \dots, x_n] = \frac{1}{||\mathbf{X}||^2}[x_1, \dots, x_n].$$ Then $$Ae = \frac{1}{||\mathbf{X}||^2}[x_1, \dots, x_n] [\varepsilon_1, \dots, \varepsilon_n]^T = \frac{1}{||\mathbf{X}||^2} \sum_{i=1}^n x_i e_i.$$ Next we have \begin{align} E[(\hat \beta - \beta)^3|\mathbf{X}] &= E[(A\varepsilon)^3|\mathbf{X}] \\ &= \sum_{i=1}^n E\bigg[\frac{(x_i\varepsilon_i)^3}{||\mathbf{X}||^6}\bigg|\mathbf{X}\bigg] \\ \end{align} where all the other terms have been dropped by assuming $$E[e_i e_j^2|\mathbf{X}] = 0$$ for $$i \neq j$$.

Then \begin{align} E[(\hat \beta - \beta)^3|\mathbf{X}] &= \frac{1}{||\mathbf{X}||^6} \sum_{i=1}^n x_i^3 E[\varepsilon_i^3|\mathbf{X}] \\ &= \mu_3 \frac{\sum_{i=1}^n x_i^3}{||\mathbf{X}||^6}. \end{align}

Similarly for $$k=4$$, if we drop terms by setting conditions on the expectation of the various error terms we get \begin{align} E[(\hat \beta - \beta)^4|\mathbf{X}] &= \mu_4 \frac{\sum_{i=1}^n x_i^4}{||\mathbf{X}||^8}. \end{align}

So assuming these expressions are correct, the pattern for the moments is clear. I now need to get the order of magnitude with respect to $$n$$ of \begin{align} E[(\hat \beta - \beta)^k] &= E[E[(\hat \beta - \beta)^k|\mathbf{X}]] \\ &= \mu_k E\bigg[\frac{\sum_{i=1}^n x_i^k}{||\mathbf{X}||^{2k}}\bigg] \\ &= \dots \end{align}

I am not sure how to evaluate this expectation and find its order of magnitude with respect to $$n$$ as $$n\to \infty$$.

• How are "$\beta_n$" and "$\hat\beta$" related? What are you willing to assume about the error distribution? (You must assume something quite specific, for otherwise almost nothing can be said about the higher moments of the estimator, any of which could be infinite or undefined.)
– whuber
Dec 14 '20 at 16:35
• I fixed the typo and added the conditions on the error distribution at the start of the post. For the higher moments my post already specified conditions on the error so I left those unchanged. Dec 14 '20 at 17:04
• Ultimately my motivation for this post is I am taking the expectation of a Taylor expansion of a well-behaved function $f(\hat \beta)$ around $\hat \beta = \beta$, and truncating at the second order: $E[f(\hat \beta)] = f(\beta) + \frac{1}{2}E[(\hat \beta - \beta)^2]f''(\beta) + \text{remainder}$. I need to be sure that it is safe to neglect the higher order moments as $n \to \infty$. While I know that $E[(\hat \beta - \beta)^2] = O(1/n)$ I have not been able to confirm the orders of the higher moments, in particular $E[(\hat \beta - \beta)^3]$. Dec 14 '20 at 17:06
• When the errors are iid Normal, that third moment is zero and the fourth moment can be computed from $X.$ But generally, as I have pointed out several times now, nothing can be said about those moments. For instance, the third moment could be infinite for all $n.$
– whuber
Dec 14 '20 at 17:46
• That's a misunderstanding: the properties of the limit of a sequence do not need to pertain to individual members of the sequence. It's easy to construct sequences of random variables that have no finite moments of any order (from the first on up) yet are asymptotically Normal. All this is easier to analyze and understand when you strip away the regression trappings from the question and focus on the core issue of relating higher moments in a sequence of random variables to lower moments.
– whuber
Dec 14 '20 at 20:21

You have not assumed enough about $$\epsilon_i$$ to determine the answer. If you only want two moments of $$\hat\beta$$ is is enough to make assumptions about two joint moments of $$\epsilon_i$$, as you have done. This is not enough for higher moments of $$\epsilon$$.

1. First, as @whuber has pointed out, you need to assume the higher moments of $$\epsilon_i$$ exist. There are distributions satisfying your assumptions that do not have a finite third moment, such as $$\epsilon_i$$ being iid $$t_3$$ variables. Existence of third moments might seem trivial, but it's not unusual for people to model long-tailed regression errors using $$t$$ distributions, and a $$t_k$$ distribution has only $$k-1$$ finite moments.

2. Second, you need control higher-order joint or conditional expectations if you're not willing to assume independence. You have assumed $$E[\epsilon_i\epsilon_j]=0$$, which gives the same variance as if the $$\epsilon_i$$ were independent. It does not give the same third moments as if the $$\epsilon_i$$ were independent, and you can't conclude, eg, that $$\left(\sum_i x_i\epsilon_i\right)^3= \sum_i (x_i\epsilon_i)^3$$ You might as well assume the $$\epsilon$$s are independent, since that's not much weaker than the assumptions you'll need about expectations of all higher-order products.

3. Similarly, you might as well assume $$\epsilon_i$$ are identically distributed, since you'll have to assume they have the same moments of all finite orders, and that's not usefully weaker.

So, if you assume $$\epsilon_i$$ are iid with finite moments of all orders, you can do the transformations you've done and we end up with the question of the size of $$\mu_k \frac{\sum x_i^k}{\|X\|^{2k}}=\mu_k \frac{\|X\|_{k}^{k}}{\|X\|_2^{2k}}$$

For fixed $$k$$, and if the expectations of powers of $$X$$ exist (you need moment assumptons on $$x$$,too), this is of order $$n^{-k}$$ if $$E[x_i]\neq 0$$ (so that $$\|X\|$$ is proportional to $$n$$) and of order $$n^{-k/2}$$ if $$E[x_i]=0$$ (so that $$\|X\|$$ is proportional to $$\sqrt{n}$$)[this is where the omission of the intercept comes back to bite you].

The dependence on $$k$$, however, can be almost arbitrarily complicated, so you need more assumptions to say anything about the moments for large $$(k, n)$$.

• This is an excellent answer thanks, its great to finally confirm that my second-order Taylor expansion will be valid once these moments are controlled with appropriate assumptions. I'm still unsure about the evaluation of $E[||X||_k^k/||X||_2^{2k}]$, how are you finding that both $||X||_2$ and $||X||_k$ are proportional to $n$ when $E[x_i] \neq 0$, and $\sqrt{n}$ when $E[x_i] = 0$ so that ultimately we get $E[||X||_k^k/||X||_2^{2k}] = O(n^k/n^{2k}) = O(n^{-k})$? Dec 15 '20 at 14:04
• ...for the $E[x_i] \neq 0$ case, and similarly we get $O(n^{-k/2})$ for the $E[x_i] = 0$ case? Dec 15 '20 at 14:35
• For $||X||_2$ I have $||X||_2^2 = \sum_{i=1}^n x_i^2 \stackrel{}{\to} n E[x_i^2]$ which implies $||X||_2 \to (n E[x_i^2])^{1/2} = O(n^{1/2})$. For $||X||_k$ I have $||X||_k^k = \sum_{i=1}^n x_i^k \stackrel{}{\to} n E[x_i^k]$ which implies $||X||_k \to (n E[x_i^k])^{1/k} = O(n^{1/k})$. Then I get $E[||X||_k^k/||X||_2^{2k}] \to E[O(n^{1/k})^{k}/O(n^{1/2})^{2k}] = O(n/n^{k}) = O(n^{-k+1})$ Dec 15 '20 at 15:01
• So are you sure the norms are proportional to $n$ in the $E[x_i] \neq 0$ case? Should the final result be $O(n^{-k+1})$ instead of $O(n^{-k})$ or have I made a mistake? Actually, I don't see how $E[x_i] = 0$ or $E[x_i] \neq 0$ comes into play, I didn't use it in the above since we are dealing with $E[x_i^k]$ where $k > 1$. Dec 15 '20 at 15:05
• A sum of $n$ independent things is of size $O_p(n)$ if they don't have mean zero (because even just the mean is that big), and a sum of $n$ independent mean-zero things is of size $O_p(\sqrt{n})$ because its variance is proportional to $n$ Dec 16 '20 at 6:08