$E[(\beta_n-\beta)^2|\mathbf{X}]=\sigma^2(\mathbf{X}^T\mathbf{X})^{-1}$, what about $E[(\beta_n-\beta)^k|\mathbf{X}]$ for $k=3,4$?

Question

Ultimately my goal with this post is to find the orders of magnitude of $E[(\beta_n-\beta)^k]$ for $k\ge 3$ as $n \to \infty$ so that I can be sure that is safe to drop higher order terms in a Taylor expansion based approximation. As a first step towards $E[(\beta_n-\beta)^k]$ I consider $E[(\beta_n-\beta)^k|\mathbf{X}]$ in what follows.

My real interest is in the case of multiple regression (including an intercept term) but I think it is easier to focus on a simpler case first. Suppose we have the simple intercept-free linear regression model $$ Y_i = \beta X_i + \varepsilon_i, \quad \quad i=1,\dots,n, $$ where $E[\varepsilon_i|\mathbf{X}] = 0$, $\text{Var}[\varepsilon_i^2|\mathbf{X}] = \sigma^2$, and $E[\varepsilon_i\varepsilon_j|\mathbf{X}] = 0$ for $i\neq j$.

It is known that $\hat \beta = \beta + A\varepsilon$ where $$ A = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'. $$ In this simple setting the data matrix $\mathbf{X}$ is actually a vector: $$ \mathbf{X} = [x_1,\dots,x_n]^T. $$ So $A\varepsilon$ is simply a scalar in our simple model.

Now consider evaluating $E[(\hat \beta - \beta)^3|\mathbf{X}]$ or $E[(\hat \beta - \beta)^4|\mathbf{X}]$ using the same process (see p. 61, Econometric Analysis 8 ed. by Green or p.29, Econometrics by Hayashi) that is used for finding an expression for $E[(\hat \beta - \beta)^2|\mathbf{X}]$. That is \begin{align} E[(\hat \beta - \beta)^2|\mathbf{X}] &= E[(A\varepsilon)^2|\mathbf{X}] \\ & = A E[\varepsilon \varepsilon^T|\mathbf{X}]] A^T \\ & = A \sigma^2 A^T \quad \quad \quad \quad \quad \quad \quad \quad \text{(using assumption $E[\varepsilon \varepsilon^T|\mathbf{X}]] = \sigma^2$ )}\\ & = \sigma^2 (\mathbf{X}^T\mathbf{X})^{-1}. \end{align}

Now lets apply this procedure to the higher moments: \begin{align} E[(\hat \beta - \beta)^3|\mathbf{X}] &= E[(A\varepsilon)^3|\mathbf{X}] \\ & = A E[\varepsilon (A \varepsilon) \varepsilon^T|\mathbf{X}]] A^T \\ \end{align}

and

\begin{align} E[(\hat \beta - \beta)^4|\mathbf{X}] &= E[(A\varepsilon)^4|\mathbf{X}] \\ & = A E[\varepsilon (A \varepsilon)^2 \varepsilon^T|\mathbf{X}]] A^T \\ \end{align}

In both cases we get stuck with $A$'s inside the expectation. So is there any way to overcome this and obtain explicit expressions for $E[(\hat \beta - \beta)^3|\mathbf{X}]$ and $E[(\hat \beta - \beta)^4|\mathbf{X}]$?} This issue is overcome in the attempt below.

Further attempt: We have $$ A = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}' = ([x_1, \dots, x_n] [x_1, \dots, x_n]^T)^{-1} [x_1, \dots, x_n] = \frac{1}{||\mathbf{X}||^2}[x_1, \dots, x_n]. $$ Then $$ Ae = \frac{1}{||\mathbf{X}||^2}[x_1, \dots, x_n] [\varepsilon_1, \dots, \varepsilon_n]^T = \frac{1}{||\mathbf{X}||^2} \sum_{i=1}^n x_i e_i. $$ Next we have $$ \begin{align} E[(\hat \beta - \beta)^3|\mathbf{X}] &= E[(A\varepsilon)^3|\mathbf{X}] \\ &= \sum_{i=1}^n E\bigg[\frac{(x_i\varepsilon_i)^3}{||\mathbf{X}||^6}\bigg|\mathbf{X}\bigg] \\ \end{align} $$ where all the other terms have been dropped by assuming $E[e_i e_j^2|\mathbf{X}] = 0$ for $i \neq j$.

Then \begin{align} E[(\hat \beta - \beta)^3|\mathbf{X}] &= \frac{1}{||\mathbf{X}||^6} \sum_{i=1}^n x_i^3 E[\varepsilon_i^3|\mathbf{X}] \\ &= \mu_3 \frac{\sum_{i=1}^n x_i^3}{||\mathbf{X}||^6}. \end{align}

Similarly for $k=4$, if we drop terms by setting conditions on the expectation of the various error terms we get \begin{align} E[(\hat \beta - \beta)^4|\mathbf{X}] &= \mu_4 \frac{\sum_{i=1}^n x_i^4}{||\mathbf{X}||^8}. \end{align}

So assuming these expressions are correct, the pattern for the moments is clear. I now need to get the order of magnitude with respect to $n$ of $$ \begin{align} E[(\hat \beta - \beta)^k] &= E[E[(\hat \beta - \beta)^k|\mathbf{X}]] \\ &= \mu_k E\bigg[\frac{\sum_{i=1}^n x_i^k}{||\mathbf{X}||^{2k}}\bigg] \\ &= \dots \end{align} $$

I am not sure how to evaluate this expectation and find its order of magnitude with respect to $n$ as $n\to \infty$.

Answer 1

You have not assumed enough about $\epsilon_i$ to determine the answer. If you only want two moments of $\hat\beta$ is is enough to make assumptions about two joint moments of $\epsilon_i$, as you have done. This is not enough for higher moments of $\epsilon$.

1. First, as @whuber has pointed out, you need to assume the higher moments of $\epsilon_i$ exist. There are distributions satisfying your assumptions that do not have a finite third moment, such as $\epsilon_i$ being iid $t_3$ variables. Existence of third moments might seem trivial, but it's not unusual for people to model long-tailed regression errors using $t$ distributions, and a $t_k$ distribution has only $k-1$ finite moments.

2. Second, you need control higher-order joint or conditional expectations if you're not willing to assume independence. You have assumed $E[\epsilon_i\epsilon_j]=0$, which gives the same variance as if the $\epsilon_i$ were independent. It does not give the same third moments as if the $\epsilon_i$ were independent, and you can't conclude, eg, that $$\left(\sum_i x_i\epsilon_i\right)^3= \sum_i (x_i\epsilon_i)^3$$ You might as well assume the $\epsilon$s are independent, since that's not much weaker than the assumptions you'll need about expectations of all higher-order products.

3. Similarly, you might as well assume $\epsilon_i$ are identically distributed, since you'll have to assume they have the same moments of all finite orders, and that's not usefully weaker.

So, if you assume $\epsilon_i$ are iid with finite moments of all orders, you can do the transformations you've done and we end up with the question of the size of $$\mu_k \frac{\sum x_i^k}{\|X\|^{2k}}=\mu_k \frac{\|X\|_{k}^{k}}{\|X\|_2^{2k}}$$

For fixed $k$, and if the expectations of powers of $X$ exist (you need moment assumptons on $x$,too), this is of order $n^{-k}$ if $E[x_i]\neq 0$ (so that $\|X\|$ is proportional to $n$) and of order $n^{-k/2}$ if $E[x_i]=0$ (so that $\|X\|$ is proportional to $\sqrt{n}$)[this is where the omission of the intercept comes back to bite you].

The dependence on $k$, however, can be almost arbitrarily complicated, so you need more assumptions to say anything about the moments for large $(k, n)$.