# Estimate the covariance matrix of a normal distribution if the mean vectors is given by a linear rule

Let $X=(x_1,\ldots,x_n)^\top\in\Bbb{R}^n$ be a random vector that follows a multivariate Gaussian distribution with known mean vector $\mu=(\mu_1,\ldots,\mu_n)^\top\in\Bbb{R}^n$. The covariance matrix $\Sigma$ is unknown, and we want to estimate it if it known that the random vector $X$ is composed as follows $$x_t=\mathbf{w}_t\cdot(\mathbf{u}+\mathbf{e})+b_t \:, \:\:\: t=1,\ldots,n$$ where $\mathbf{w}_t,\mathbf{u}, \mathbf{e}\in\Bbb{R}^m$, $b_t\in\Bbb{R}$. Moreover, $\mathbf{e}$ is a random vector such that $\mathbf{e}\sim N(0,\sigma I_m)$, where $I_m$ denotes the unitary matrix of order $m$.

Using the definition of the covariance matrix $\Sigma=\big(\sigma_{ij}\big)_{i,j=1}^n$, we have: $$\Sigma = E\Big[(X-\mu)(X-\mu)^\top\Big],$$ or $$\sigma_{ij} = E\Big[(x_i-\mu_i)(x_j-\mu_j)\Big] = E\Big[x_ix_j\Big]-\mu_i\mu_j.$$ We need to find $E\Big[x_ix_j\Big]$. To this end, we write $$x_ix_j = [\mathbf{w}_i\cdot(\mathbf{u}+\mathbf{e})+b_i] [\mathbf{w}_j\cdot(\mathbf{u}+\mathbf{e})+b_j],$$ which, after some easy manipulations, yields $$x_ix_j = (\mathbf{w}_i\mathbf{u})(\mathbf{w}_j\mathbf{u}) + (\mathbf{w}_i\mathbf{u})(\mathbf{w}_j\mathbf{e}) + (\mathbf{w}_j\mathbf{u})(\mathbf{w}_i\mathbf{e}) + (\mathbf{w}_i\mathbf{e})(\mathbf{w}_j\mathbf{e}) + b_j\mathbf{w}_i\cdot\mathbf{u} + b_j\mathbf{w}_i\cdot\mathbf{e} + b_i\mathbf{w}_j\cdot\mathbf{u} + b_i\mathbf{w}_j\cdot\mathbf{e} + b_ib_j,$$ and, by applying the expected value operator, we have $$E\Big[x_ix_j\Big] = (\mathbf{w}_i\cdot\mathbf{u})(\mathbf{w}_j\cdot\mathbf{u}) + b_i\mathbf{w}_j\cdot\mathbf{u} + b_j\mathbf{w}_i\cdot\mathbf{u} + b_ib_j + E\Big[(\mathbf{w}_i\cdot\mathbf{e})(\mathbf{w}_j\cdot\mathbf{e})\Big].$$ Now, we write $$E\Big[(\mathbf{w}_i\cdot\mathbf{e})(\mathbf{w}_j\cdot\mathbf{e})\Big] = E\Big[(\mathbf{w}_i^\top\mathbf{e})(\mathbf{w}_j^\top\mathbf{e})\Big] = E\Big[\mathbf{w}_i^\top(\mathbf{e}\mathbf{e}^\top)\mathbf{w}_j\Big] = \mathbf{w}_i^\top(\sigma I_m)\mathbf{w}_j = \sigma\mathbf{w}_i^\top\mathbf{w}_j = \sigma\mathbf{w}_i\cdot\mathbf{w}_j.$$

Finally, $$\sigma_{ij} = (\mathbf{w}_i\cdot\mathbf{u})(\mathbf{w}_j\cdot\mathbf{u}) + b_i\mathbf{w}_j\cdot\mathbf{u} + b_j\mathbf{w}_i\cdot\mathbf{u} + b_ib_j + \sigma\mathbf{w}_i\cdot\mathbf{w}_j - \mu_i\mu_j,$$ or, $$\sigma_{ij} = \mathbf{w}_i^\top\mathbf{u}\mathbf{u}^\top\mathbf{w}_j + b_i\mathbf{w}_j^\top\mathbf{u} + b_j\mathbf{w}_i^\top\mathbf{u} + b_ib_j + \sigma\mathbf{w}_i^\top\mathbf{w}_j - \mu_i\mu_j.$$

May you confirm if this is correct, or suggest your corrections? Thanks!

EDIT Actually, despite the fact the all the above are technically correct, the answer is not precise...

We should not forget that $\mu_t=\mathbf{w}_t\cdot\mathbf{u}+b_t$, so the final result is given by $$\sigma_{ij}=\sigma\mathbf{w}_i\cdot\mathbf{w}_j.$$

• How does the "linear rule" help, since we already know the mean-values vector? – Alecos Papadopoulos Sep 22 '14 at 15:50
• @AlecosPapadopoulos, actually, you're perfectly right... I have made some major changes in my original post. Could you please check if now is correct? Thanks a lot! – nullgeppetto Sep 24 '14 at 7:02
• You should clarify whether the vectors are initially row- or column. Also, in the initial expression for the (univariate) $x_t$, either $w$ or $u,e$ should be transposed, otherwise they are not conformable for multiplication. You should check whether this changes the end result or not. – Alecos Papadopoulos Sep 24 '14 at 12:32
• @AlecosPapadopoulos, thanks for the response, but I thought the notation would be clear enough. I write, for instance, $\mu=(\mu_1,\ldots,\mu_n)^\top$, so I though it would be clear that $\mu$ is a column vector. But, your are right, it may not be clear for the rest of the vectors being used here. So, yes, all of the used vectors are column real vectors, while I use interchangeably the inner product operator both using $\cdot$ between vectors, or the transpose notation ($^\top$) on the first vector, as shown in my original post. Isn't it clear? – nullgeppetto Sep 24 '14 at 12:40