# Expected Value of Quadratic Form [duplicate]

On page 9 of Linear Regression Analysis 2nd Edition of Seber and Lee there is a proof for the expected value of a quadratic form that I don't understand.

Let $$X = (X_i)$$ be an $$n \times 1$$ random vector and let $$A$$ be an $$n \times n$$ symmetric matrix. If $$\mathbb{E}[X] = \mu$$ and $$\operatorname{Var}[X] = \Sigma = (\sigma_{ij})$$ then $$\mathbb{E}[X^T AX] = \operatorname{tr}(A\Sigma) + \mu^T A\mu$$

The problem I have is almost right out the gate, I can't see how $$\mathbb{E}[X^T AX] = \operatorname{tr}(\mathbb{E}[X^T AX])$$ I think I get the rest of the proof, but if someone here can't point me in the right direction on this part, I'd be forever grateful!

• It is not $\mathbb{E}[X^{T}AX] = tr(\mathbb{E}[X^{T}AX]$ that you need to be concerned about but the trace of $A\Sigma$ which is a $n\times n$ matrix. See my answer for a complete derivation. Jonathan Christensen's remarks about $\text{tr}([3])$ and scalars versus matrices are not applicable to this problem. Jan 19, 2013 at 3:19
• @DilipSarwate, you are obviously unfamiliar with the matrix algebra version of this proof (Wikipedia has a very short outline), which is what Kyle is asking about here. You present an alternative proof, but your answer doesn't address the original question at all. Jan 19, 2013 at 17:42
• @JonathanChristensen The matrix algebra version of the proof needs knowledge such as $E$ and $\text{tr}$ are commutative linear operators, and is a bad example to learn from. The expectation of a matrix $B$ (with random variables as entries) is denoted $E[B]$ and is simply the matrix of expected values. In general, the result $E[B]= \text{tr}(E[B])$ is false since the left side is a matrix and the right side a scalar or $1\times 1$ matrix if you will. And the result holds exactly when $B$ is a $1\times 1$ matrix in which case the trace operation on the right is an identity map. Jan 19, 2013 at 20:19
• @DilipSarwate Whatever you think of the pedagogical merits of the matrix algebra version of the prove, that is the question that Kyle asked. I answered it. You ignored it and made snide comments. Jan 19, 2013 at 22:33
• Did you like the book? I would like to learn more about the algebra of random matrices/vectors.
– Joe
Oct 7, 2020 at 18:28

As Jonathan Christensen points out, $X^TAX$ is a $1\times 1$ matrix; in fact, it is the (univariate) random variable $$X^TAX = \sum_{i=1}^n \sum_{j=1}^n a_{i,j}X_iX_j.$$ So what is its expectation? Clearly we have \begin{align*} E[X^TAX] &= E\left[\sum_{i=1}^n \sum_{j=1}^n a_{i,j}X_iX_j\right]\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}E[X_iX_j] & \text{by linearity of expectation}\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}(\sigma_{i,j}+\mu_i\mu_j) &\text{apply covariance formula}\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}\sigma_{j,i} +\sum_{i=1}^n \sum_{j=1}^n a_{i,j}\mu_i\mu_j &\text{since}~\Sigma~\text{is a symmetric matrix}\\ &= \sum_{i=1}^n [A\Sigma]_{i,i} + \mu^TA\mu\\ &= \text{tr}(A\Sigma) + \mu^TA\mu \end{align*}

• This is a fine proof of the result, but it doesn't address Kyle's actual question. Jan 19, 2013 at 14:42
• This expansion into a double sum helped nail for me why the trace trick is needed, and why $\mathbb{E}_{x \sim \mathcal{N}(\mu, \Sigma)} \left[ (x-\mu)^T \Sigma^{-1} (x-\mu) \right]$ isn't just zero. Thank you. Dec 22, 2015 at 19:05

Since $X$ is an $n\times1$ vector, $\mathbb E[X^{T}AX]$ is a $1\times1$ matrix. The trace is the sum of diagonal entries, but $\mathbb E[X^TAX]$ only has one entry, so its trace is simply equal to that one entry. If we consider a $1\times1$ matrix to be equivalent to a scalar, then the equality you're worried about follows.

Basically, what's $tr([3])$? It's obviously 3. Now, you might argue that strictly speaking $[3] \neq 3$, because one is a matrix and the other is a real number, but they're basically equivalent, and if we're a bit loose with notation then we can say they're equal.

• Haha, Damn... I knew that the quadratic form was a scalar, but it seemed so weird to take the trace of a scalar that I thought that I missed something. Thanks a bunch.
– Kyle
Jan 19, 2013 at 1:49
• I am not sure what this answer means. The trace that is being computed is the trace of $A\Sigma$ which is not a scalar. Jan 19, 2013 at 3:04
• @DilipSarwate You didn't read the question carefully, did you? Quote: "I can't see how $\mathbb{E}[X^{T}AX] = tr(\mathbb{E}[X^{T}AX])$. I think I get the rest of the proof [...]" The part Kyle was confused about was exactly the part I addressed. Jan 19, 2013 at 14:40
• The final bit of manipulation that Jonathan might want to mention is that ${\rm tr}AB = {\rm tr}BA$ even though these matrices may be of different size. So ${\rm tr}X'AX = {\rm tr} AXX'$ at which point the constant $A$ can be pulled through the expectation sign. May 31, 2013 at 14:20
• @StasK: You have identified the entire point of taking the trace in the first place. With Jonathan's explanation, this is a full and complete answer (despite some contrary claims in comments to the question).
– whuber
May 31, 2013 at 14:28

To get the intuition behind it, recall that when $X$ were a univariate random variable it follows that $$E[a^{2}X^{2}]=a^{2}Var(X)+E[aX]^{2}$$

On the other hand, when $X$ is a random vector, let $A = C^\prime C$, such that $X^\prime AX= X^{\prime}C^{\prime}CX$, where $C$ denotes the square root of $A$. This assumes that $A$ is positive definite - this assumption, however, is imposed for the sake of illustration. It follows that $$E[X^{\prime}C^{\prime}CX]=Var(CX)+E[CX]^{\prime}E[CX]$$ and thus $$E[X^{\prime}AX]=C^{\prime}Var(X)C+E[X]^{\prime}AE[X]$$

Finally, one can represent $C^{\prime} \Sigma C$ as $tr(A\Sigma)$, since $A=C^\prime C$. Obviously, the general property does not require knowing the square root of $A$. However, one can see where this property comes from.