12
$\begingroup$

On page 9 of Linear Regression Analysis 2nd Edition of Seber and Lee there is a proof for the expected value of a quadratic form that I don't understand.

Let $X = (X_i)$ be an $n \times 1 $ random vector and let $A$ be an $ n \times n$ symmetric matrix. If $\mathbb{E}[X] = \mu$ and $\operatorname{Var}[X] = \Sigma = (\sigma_{ij})$ then $\mathbb{E}[X^T AX] = \operatorname{tr}(A\Sigma) + \mu^T A\mu$

The problem I have is almost right out the gate, I can't see how $\mathbb{E}[X^T AX] = \operatorname{tr}(\mathbb{E}[X^T AX])$ I think I get the rest of the proof, but if someone here can't point me in the right direction on this part, I'd be forever grateful!

$\endgroup$
5
  • $\begingroup$ It is not $\mathbb{E}[X^{T}AX] = tr(\mathbb{E}[X^{T}AX]$ that you need to be concerned about but the trace of $A\Sigma$ which is a $n\times n$ matrix. See my answer for a complete derivation. Jonathan Christensen's remarks about $\text{tr}([3])$ and scalars versus matrices are not applicable to this problem. $\endgroup$ Jan 19, 2013 at 3:19
  • $\begingroup$ @DilipSarwate, you are obviously unfamiliar with the matrix algebra version of this proof (Wikipedia has a very short outline), which is what Kyle is asking about here. You present an alternative proof, but your answer doesn't address the original question at all. $\endgroup$ Jan 19, 2013 at 17:42
  • 1
    $\begingroup$ @JonathanChristensen The matrix algebra version of the proof needs knowledge such as $E$ and $\text{tr}$ are commutative linear operators, and is a bad example to learn from. The expectation of a matrix $B$ (with random variables as entries) is denoted $E[B]$ and is simply the matrix of expected values. In general, the result $E[B]= \text{tr}(E[B])$ is false since the left side is a matrix and the right side a scalar or $1\times 1$ matrix if you will. And the result holds exactly when $B$ is a $1\times 1$ matrix in which case the trace operation on the right is an identity map. $\endgroup$ Jan 19, 2013 at 20:19
  • $\begingroup$ @DilipSarwate Whatever you think of the pedagogical merits of the matrix algebra version of the prove, that is the question that Kyle asked. I answered it. You ignored it and made snide comments. $\endgroup$ Jan 19, 2013 at 22:33
  • $\begingroup$ Did you like the book? I would like to learn more about the algebra of random matrices/vectors. $\endgroup$
    – Joe
    Oct 7, 2020 at 18:28

3 Answers 3

25
$\begingroup$

As Jonathan Christensen points out, $X^TAX$ is a $1\times 1$ matrix; in fact, it is the (univariate) random variable $$X^TAX = \sum_{i=1}^n \sum_{j=1}^n a_{i,j}X_iX_j.$$ So what is its expectation? Clearly we have $$\begin{align*} E[X^TAX] &= E\left[\sum_{i=1}^n \sum_{j=1}^n a_{i,j}X_iX_j\right]\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}E[X_iX_j] & \text{by linearity of expectation}\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}(\sigma_{i,j}+\mu_i\mu_j) &\text{apply covariance formula}\\ &= \sum_{i=1}^n \sum_{j=1}^n a_{i,j}\sigma_{j,i} +\sum_{i=1}^n \sum_{j=1}^n a_{i,j}\mu_i\mu_j &\text{since}~\Sigma~\text{is a symmetric matrix}\\ &= \sum_{i=1}^n [A\Sigma]_{i,i} + \mu^TA\mu\\ &= \text{tr}(A\Sigma) + \mu^TA\mu \end{align*}$$

$\endgroup$
2
  • 2
    $\begingroup$ This is a fine proof of the result, but it doesn't address Kyle's actual question. $\endgroup$ Jan 19, 2013 at 14:42
  • 1
    $\begingroup$ This expansion into a double sum helped nail for me why the trace trick is needed, and why $\mathbb{E}_{x \sim \mathcal{N}(\mu, \Sigma)} \left[ (x-\mu)^T \Sigma^{-1} (x-\mu) \right]$ isn't just zero. Thank you. $\endgroup$ Dec 22, 2015 at 19:05
11
$\begingroup$

Since $X$ is an $n\times1$ vector, $\mathbb E[X^{T}AX]$ is a $1\times1$ matrix. The trace is the sum of diagonal entries, but $\mathbb E[X^TAX]$ only has one entry, so its trace is simply equal to that one entry. If we consider a $1\times1$ matrix to be equivalent to a scalar, then the equality you're worried about follows.

Basically, what's $tr([3])$? It's obviously 3. Now, you might argue that strictly speaking $[3] \neq 3$, because one is a matrix and the other is a real number, but they're basically equivalent, and if we're a bit loose with notation then we can say they're equal.

$\endgroup$
5
  • $\begingroup$ Haha, Damn... I knew that the quadratic form was a scalar, but it seemed so weird to take the trace of a scalar that I thought that I missed something. Thanks a bunch. $\endgroup$
    – Kyle
    Jan 19, 2013 at 1:49
  • $\begingroup$ I am not sure what this answer means. The trace that is being computed is the trace of $A\Sigma$ which is not a scalar. $\endgroup$ Jan 19, 2013 at 3:04
  • 2
    $\begingroup$ @DilipSarwate You didn't read the question carefully, did you? Quote: "I can't see how $\mathbb{E}[X^{T}AX] = tr(\mathbb{E}[X^{T}AX])$. I think I get the rest of the proof [...]" The part Kyle was confused about was exactly the part I addressed. $\endgroup$ Jan 19, 2013 at 14:40
  • 2
    $\begingroup$ The final bit of manipulation that Jonathan might want to mention is that ${\rm tr}AB = {\rm tr}BA$ even though these matrices may be of different size. So ${\rm tr}X'AX = {\rm tr} AXX'$ at which point the constant $A$ can be pulled through the expectation sign. $\endgroup$
    – StasK
    May 31, 2013 at 14:20
  • 4
    $\begingroup$ @StasK: You have identified the entire point of taking the trace in the first place. With Jonathan's explanation, this is a full and complete answer (despite some contrary claims in comments to the question). $\endgroup$
    – whuber
    May 31, 2013 at 14:28
1
$\begingroup$

To get the intuition behind it, recall that when $X$ were a univariate random variable it follows that \begin{equation} E[a^{2}X^{2}]=a^{2}Var(X)+E[aX]^{2} \end{equation}

On the other hand, when $X$ is a random vector, let $A = C^\prime C$, such that $X^\prime AX= X^{\prime}C^{\prime}CX$, where $C$ denotes the square root of $A$. This assumes that $A$ is positive definite - this assumption, however, is imposed for the sake of illustration. It follows that \begin{equation} E[X^{\prime}C^{\prime}CX]=Var(CX)+E[CX]^{\prime}E[CX] \end{equation} and thus \begin{equation} E[X^{\prime}AX]=C^{\prime}Var(X)C+E[X]^{\prime}AE[X] \end{equation}

Finally, one can represent $C^{\prime} \Sigma C$ as $tr(A\Sigma)$, since $A=C^\prime C$. Obviously, the general property does not require knowing the square root of $A$. However, one can see where this property comes from.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.