Covariance matrix of complex random variables

Question

Let's say we have datapoints $x_i \in \mathbb{C}^N$, like this: $x_1 = \begin{pmatrix}z_1 & z_2 & ... & z_N \end{pmatrix}^T$ and $x_2=\begin{pmatrix}z_1 & z_2 & ... & z_N \end{pmatrix}^T$. My question is, how to interpret the covariance matrix calculated for the data points $x_i \in \mathbb{C}^N$. I know how to interpret the real-valued diagonal, which is just the variance of the corresponding variable, but the off-diagonal elements are complex numbers. Can someone give an intuition of what these represent. For example the covariance between $z_1$ and $z_2$ yields the number $a+bi$. Then, does $a$ represent the covariance of the real parts of $z_1$ and $z_2$ and $b$ the covariance of the imaginary parts of $z_1$ and $z_2$?

Answer 1

Here is a geometric interpretation.

First, take two vectors in $\mathbb{R}^2$

$$\vec{\mathbb{z}}=[x,y] \,, \vec{\mathbb{w}}=[u,v]$$

For these vectors, there are two standard types of "products", the dot product $$\vec{\mathbb{z}}\dot{}\vec{\mathbb{w}}=xu+yv$$ and the cross product* $$\vec{\mathbb{z}}\times\vec{\mathbb{w}}=xv-yu$$ which can be interpreted as $$\vec{\mathbb{z}}\dot{}\vec{\mathbb{w}}_\perp$$ where $\vec{\mathbb{w}}_\perp=[v,-u]$ has the same magnitude as $\vec{\mathbb{w}}$ but is orthogonal.

(*Technically this "2D cross product" is defined as $[0,0,\vec{\mathbb{z}}\times\vec{\mathbb{w}}]\equiv[x,y,0]\times[u,v,0]$.)

In terms of geometric intuition, the dot product between two vectors measures how well they align (think correlation), but also their relative magnitudes (think standard deviations), i.e. $$\vec{\mathbb{z}}\dot{}\vec{\mathbb{w}}=||\vec{\mathbb{z}}||\,||\vec{\mathbb{w}}||\,\cos[\theta]$$ where $\theta$ is the angle between them (compare to $\sigma_{xy}=\sigma_x\sigma_y\rho_{xy}$). Note that the dot product can also be written as $\mathbb{z}^T\mathbb{w}$, where $\mathbb{z}$ and $\mathbb{w}$ are just $\vec{\mathbb{z}}$ and $\vec{\mathbb{z}}$ written as column vectors.

Now let us do the same thing with two scalars in the complex plane ($z,w\in\mathbb{C}$), i.e. $$z=x+iy \,, w=u+iv$$ What is the equivalent to the "dot product" here? It is actually the same as above, but now using the conjugate transpose, i.e. $z^*\equiv\bar{z}^T$ (also written as $z^\dagger$).

Since the transpose of a scalar is just that same scalar, the complex dot product is then $$z^{\dagger}w=\bar{z}w=(x-iy)(u+iv)=(xu+yv)+i(xv-yu)$$ We can immediately notice two things. First, the complex dot product is equivalent to $$z^{\dagger}w=(\vec{\mathbb{z}}\dot{}\vec{\mathbb{w}})+i(\vec{\mathbb{z}}\dot{}\vec{\mathbb{w}}_\perp)$$ i.e. it is a complex number whose real component is the dot product of the corresponding 2-vectors, and whose imaginary component is their cross product. Second, since $\bar{x}=x$ for $x\in\mathbb{R}$, the vector dot product we started with can be written as $\vec{\mathbb{z}}\dot{}\vec{\mathbb{w}}=\mathbb{z}^{\dagger}\mathbb{w}$ (i.e. we were really using the conjugate transpose all along).

Now for the covariance matrix.

For simplicity, let us assume that all random variables have zero mean. Then the covariance is defined as $$\mathrm{Cov}[z,w]\equiv\mathbb{E}[\bar{z}w]$$ so we have \begin{align} \mathrm{Cov}[z,w] &= \mathrm{Re}[\sigma_{z,w}]+i\,\mathrm{Im}[\sigma_{z,w}] \\ &= \,\mathbb{E}[\,\vec{\mathbb{z}}\dot{}\vec{\mathbb{w}}\,]+i\,\mathbb{E}[\,\vec{\mathbb{z}}\dot{}\vec{\mathbb{w}}_\perp] \end{align} The real (imaginary) component of $\sigma_{z,w}$ is the expected value of the dot (cross) product of the associated vectors $\vec{\mathbb{z}}$ and $\vec{\mathbb{w}}$.

This is the main intuition. The rest of this answer is just for completeness.

If our random variable is a column vector $\mathbb{z}=[z_1,\ldots,z_n]\in\mathbb{C}^n$ with covariance matrix $\boldsymbol{\Sigma}\in\mathbb{C}^{n\times n}$, then we have $$\Sigma_{ij}=\mathrm{Cov}[z_i,z_j]$$

Finally, if we have $m$ samples of the random variable $\mathbb{z}$, arranged as the rows of a data matrix $\boldsymbol{Z}\in\mathbb{C}^{m\times n}$, then the sample* covariance can be approximated by $$\boldsymbol{\Sigma}\approx\tfrac{1}{m}\boldsymbol{Z}^{\dagger}\boldsymbol{Z}$$ (*yes, I divided by $m$, so you can call it the "biased" sample covariance if you must.)

Answer 2

No a and b from $a+bi$ are not the real and imaginary covariances because there is also a cross term.

Writing out using the definition of complex covariance gives:

$$cov(x_1,x_2)=E[(x_1-\mu_1)(x_2-\mu_2)^\dagger]$$

$$=\frac{1}{N}\sum_{i=1}^N (x_{1,i}-\mu_1)(x_{2,i}-\mu_2)^{\ast}$$

$$=\frac{1}{N}\sum_{i=1}^N (a_{1,i}+ib_{1,i}-\mu_1^{real}-i\mu_1^{imag})(a_{2,i}+ib_{2,i}-\mu_2^{real}-i\mu_2^{imag})^{\ast}$$

$$=\frac{1}{N}\sum_{i=1}^N (a_{1,i}+ib_{1,i}-\mu_1^{real}-i\mu_1^{imag})(a_{2,i}-ib_{2,i}-\mu_2^{real}+i\mu_2^{imag})$$

$$=\frac{1}{N}\sum_{i=1}^N ((a_{1,i}-\mu_1^{real})(a_{2,i}-\mu_2^{real})-i(a_{1,i}-\mu_1^{real})(b_{2,i}-\mu_2^{imag})+i(b_{1,i}-\mu_1^{imag})(a_{2,i}-\mu_2^{real})+(b_{1,i}-\mu_1^{imag})(b_{2,i}-\mu_2^{imag}))$$

$$=\frac{1}{N}\sum_{i=1}^N ((a_{1,i}-\mu_1^{real})(a_{2,i}-\mu_2^{real})-i(a_{1,i}-\mu_1^{real})(b_{2,i}-\mu_2^{imag})+i(b_{1,i}-\mu_1^{imag})(a_{2,i}-\mu_2^{real})+(b_{1,i}-\mu_1^{imag})(b_{2,i}-\mu_2^{imag}))$$

$$=cov(a_1,a_2)+cov(b_1,b_2)-\sum_{i=1}^Ni((a_{1,i}-\mu_1^{real})(b_{2,i}-\mu_2^{imag})-(b_{1,i}-\mu_1^{imag})(a_{2,i}-\mu_2^{real}))$$

$$=cov(a_1,a_2)+cov(b_1,b_2)-i(cov(a_1,b_2)-cov(b_1,a_2))$$

Therefore, $a=cov(a_1,a_2)+cov(b_1,b_2)$ and $b=-cov(a_1,b_2)+cov(b_1,a_2)$ in $a+bi$.

The intuition for this is that the angle of the complex covariance is an unbiased estimate of the mean phase difference between the 2 distributions and the amplitude is a measure (biased) of how well the phasors cluster around this mean.

If you reverse the order ($cov(x_2,x_1)$ instead of $cov(x_1,x_2)$) then the only thing that will change is that the mean angle of the phase difference between the two distributions will be flipped onto the other side of the real axis. These terms will be in the lower right triangular part of the matrix making the complex covariance matrix Hermitian and therefore the means of each column and row vector will be a real number.