Suppose I have $n$-x-$1$ independent random variables $\mathbf{a}$ and $\mathbf{b}$ with known covariances matrices $A$ and $B$. Is it possible to express covariance matrix $C$ of random variable $\mathbf{c} = \mathbf{a} \otimes \mathbf{b}$ in terms of $A$ and $B$?

The end-goal I'm trying to achieve is to do some transformation to a sample of [$\mathbf{a}$, $\mathbf{b}$] to make sure that $\mathbf{c}$ computed on that sample is whitened. Simply whitening $\mathbf{a}$ and $\mathbf{b}$ in that sample seems not sufficient.


Example 1:

Assuming without loss of generality your vectors are centered, or that $E[\mathbf{a}] = E[\mathbf{b}] = \mathbf{0}$, the covariance matrix is \begin{align*} E[\mathbf{c}\mathbf{c}^T] &= E[(\mathbf{a}\otimes \mathbf{b})(\mathbf{a} \otimes \mathbf{b})^T] \\ &= E\left\{ \left[ \begin{array}{c} a_1\mathbf{b} \\ a_2\mathbf{b} \\ \vdots \\ a_m\mathbf{b} \end{array}\right] \left[a_1\mathbf{b}^T, a_2\mathbf{b}^T, \cdots, a_m\mathbf{b}^T \right] \right\} \\ &= \left[\begin{array}{cccc} E[a_1a_1\mathbf{b}\mathbf{b}^T] & E[a_1a_2 \mathbf{b}\mathbf{b}^T] & \cdots & E[a_1a_m \mathbf{b}\mathbf{b}^T]\\ \vdots &\vdots & \ddots & \ddots \\ E[a_ma_1 \mathbf{b}\mathbf{b}^T] & E[a_m a_2 \mathbf{b}\mathbf{b}^T] & \cdots & E[a_ma_m \mathbf{b}\mathbf{b}^T] \end{array} \right]\\ &= \left[\begin{array}{cccc} E[a_1a_1]E[\mathbf{b}\mathbf{b}^T] & E[a_1a_2]E[ \mathbf{b}\mathbf{b}^T] & \cdots & E[a_1a_m]E[ \mathbf{b}\mathbf{b}^T]\\ \vdots &\vdots & \ddots & \ddots \\ E[a_ma_1]E[ \mathbf{b}\mathbf{b}^T] & E[a_m a_2]E[ \mathbf{b}\mathbf{b}^T] & \cdots & E[a_ma_m]E[ \mathbf{b}\mathbf{b}^T] \end{array} \right]\\ &=E[\mathbf{a}\mathbf{a}'] \otimes E[ \mathbf{b}\mathbf{b}^T]. \end{align*} Notice the assumption of independence is used in the second to last equality. But we can see here that the variance of the Kronecker product is the Kronecker product of the variances.

Example 2:

Your example in the (now-deleted) comments was an example where the two vectors were not independent. In that case, the above quantity would simplify to

\begin{align*} \left[\begin{array}{cccc} E[a_1a_1\mathbf{a}\mathbf{a}^T] & E[a_1a_2 \mathbf{a}\mathbf{a}^T] & \cdots & E[a_1a_m \mathbf{a}\mathbf{a}^T]\\ \vdots &\vdots & \ddots & \ddots \\ E[a_ma_1 \mathbf{a}\mathbf{a}^T] & E[a_m a_2 \mathbf{a}\mathbf{a}^T] & \cdots & E[a_ma_m \mathbf{a}\mathbf{a}^T] \end{array} \right]. \end{align*}

The only way for this to be white would be if $E[a_i^4]=1$ for all $i=1,\ldots,m$ and $E[a_i a_j a_k a_l]=0$ if there is one pair $(m,n)$ such that $m \neq n$. Is this possible? Well if we have one distinct index, the centering makes that term $0$, but..

The problem is the terms of the form $E[a_i^2 a_j^2]$. Even if you assume your vectors are centered and white, $$ E[a_i^2 a_j^2] = \text{Var}(a_i)\text{Var}(a_j) \ge 0. $$ The only way for these terms to be zero is if one of them is a degenerate random variable.

  • $\begingroup$ I've updated my example to take independent samples and whiten them, and the outer product of examples is still not whitened -- wolframcloud.com/objects/c1640d4c-bdab-4270-b7ca-7e6813219476 $\endgroup$ – Yaroslav Bulatov Apr 5 '17 at 3:29
  • $\begingroup$ We can say WLOG that those vectors are independent (since we can whiten a,b jointly, which will make them independent). But it looks like this doesn't whiten kronecker product either $\endgroup$ – Yaroslav Bulatov Apr 5 '17 at 4:34
  • 2
    $\begingroup$ @YaroslavBulatov yes it does. If they are independent, then the covariance of their kronecker is the kronecker of their covariances. That will be identity. $\endgroup$ – Taylor Apr 5 '17 at 4:42
  • $\begingroup$ Sorry, my question may have been unclear. I have a sample of [a_i,b_i] pairs, and I want to make empirical covariance transformed sample [a_i x b_i] into identity where x is Kronecker $\endgroup$ – Yaroslav Bulatov Apr 5 '17 at 16:31
  • 1
    $\begingroup$ @muffin1974 no, but there is no loss of generality. For example, if you were considering two vectors with non-zero means, you could just replace everything with the de-meaned vector. $\endgroup$ – Taylor Dec 3 '17 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.