Expectation of Quadratic Form with Two Random Vectors Assume I have two independent $(N \times 1)$ random vectors, $\epsilon_{1} \sim N(0,\Sigma_1)$ and $\epsilon_{2} \sim N(0,\Sigma_2)$.
We could assume $\Sigma_1=\Sigma_2$ for my purposes but a general answer would be better.
What is the solution then of
$\mathbb{E}\big[\epsilon_1^{\prime} A \epsilon_2 \epsilon_2^{\prime}A^{\prime}\epsilon_1\big],$
where $A$ is any $(N\times N)$ matrix?
 A: Let's start with what's well known: when $B=(b_{ij})$ is any square matrix and $x$ is a zero-mean vector with covariance matrix $\mathbb{E}(xx^\prime)=\Sigma,$ then the definition of matrix multiplication and linearity of expectation imply
$$\mathbb{E}(x^\prime B x) = \mathbb{E}\left(\sum_{i,j} x_i b_{ij} x_j\right) = \sum_i \sum_j b_{ij}\mathbb{E}(x_ix_j) = \sum_i (B\Sigma^\prime)_{ii} = \operatorname{Tr}(B\Sigma^\prime).$$
In the circumstances, linearity of expectation and the independence of the $\epsilon_i$ reduce the problem to the previous result with $B=A\Sigma_2A^\prime$ and $x=\epsilon_1:$
$$\eqalign{
\mathbb{E}\left(\epsilon_1^\prime A \epsilon_2 \epsilon_2^\prime A^\prime \epsilon_1\right) &= \mathbb{E}\left(\mathbb{E}\left(\epsilon_1^\prime A \epsilon_2 \epsilon_2^\prime A^\prime \epsilon_1 \mid \epsilon_1\right)\right) \\
&= \mathbb{E}\left(\epsilon_1^\prime A\mathbb{E}\left( \epsilon_2 \epsilon_2^\prime \right) A^\prime \epsilon_1\right) \\
&= \mathbb{E}\left(\epsilon_1^\prime A\Sigma_2 A^\prime \epsilon_1\right) \\
&= \operatorname{Tr}\left(A\Sigma_2 A^\prime\ \Sigma_1^\prime\right).
}$$
You are free to transpose either of the $\Sigma_i$ or not, because--as covariance matrices--they are symmetric.
