Expected Value of Quadratic Form [duplicate]

Question

On page 9 of Linear Regression Analysis 2nd Edition of Seber and Lee there is a proof for the expected value of a quadratic form that I don't understand.

Let $X = (X_i)$ be an $n \times 1 $ random vector and let $A$ be an $ n \times n$ symmetric matrix. If $\mathbb{E}[X] = \mu$ and $\operatorname{Var}[X] = \Sigma = (\sigma_{ij})$ then $\mathbb{E}[X^T AX] = \operatorname{tr}(A\Sigma) + \mu^T A\mu$

The problem I have is almost right out the gate, I can't see how $\mathbb{E}[X^T AX] = \operatorname{tr}(\mathbb{E}[X^T AX])$ I think I get the rest of the proof, but if someone here can't point me in the right direction on this part, I'd be forever grateful!

Answer 1

As Jonathan Christensen points out, $X^TAX$ is a $1\times 1$ matrix; in fact, it is the (univariate) random variable $$X^TAX = \sum_{i=1}^n \sum_{j=1}^n a_{i,j}X_iX_j.$$ So what is its expectation? Clearly we have $$\begin{align*} E[X^TAX] &= E\left[\sum_{i=1}^n \sum_{j=1}^n a_{i,j}X_iX_j\right]\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}E[X_iX_j] & \text{by linearity of expectation}\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}(\sigma_{i,j}+\mu_i\mu_j) &\text{apply covariance formula}\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}\sigma_{j,i} +\sum_{i=1}^n \sum_{j=1}^n a_{i,j}\mu_i\mu_j &\text{since}~\Sigma~\text{is a symmetric matrix}\\ &= \sum_{i=1}^n [A\Sigma]_{i,i} + \mu^TA\mu\\ &= \text{tr}(A\Sigma) + \mu^TA\mu \end{align*}$$

Answer 2

Since $X$ is an $n\times1$ vector, $\mathbb E[X^{T}AX]$ is a $1\times1$ matrix. The trace is the sum of diagonal entries, but $\mathbb E[X^TAX]$ only has one entry, so its trace is simply equal to that one entry. If we consider a $1\times1$ matrix to be equivalent to a scalar, then the equality you're worried about follows.

Basically, what's $tr([3])$? It's obviously 3. Now, you might argue that strictly speaking $[3] \neq 3$, because one is a matrix and the other is a real number, but they're basically equivalent, and if we're a bit loose with notation then we can say they're equal.

Answer 3

To get the intuition behind it, recall that when $X$ were a univariate random variable it follows that \begin{equation} E[a^{2}X^{2}]=a^{2}Var(X)+E[aX]^{2} \end{equation}

On the other hand, when $X$ is a random vector, let $A = C^\prime C$, such that $X^\prime AX= X^{\prime}C^{\prime}CX$, where $C$ denotes the square root of $A$. This assumes that $A$ is positive definite - this assumption, however, is imposed for the sake of illustration. It follows that \begin{equation} E[X^{\prime}C^{\prime}CX]=Var(CX)+E[CX]^{\prime}E[CX] \end{equation} and thus \begin{equation} E[X^{\prime}AX]=C^{\prime}Var(X)C+E[X]^{\prime}AE[X] \end{equation}

Finally, one can represent $C^{\prime} \Sigma C$ as $tr(A\Sigma)$, since $A=C^\prime C$. Obviously, the general property does not require knowing the square root of $A$. However, one can see where this property comes from.