Suppose $\mathbf{Y}$ is a $n$-dimensional random vector, $A$ is a fixed $r \times n$ matrix, and $b$ is a fixed vector in $\mathbb{R}^n$. I have proven already that $$\mathbb{E}\left[A\mathbf{Y}+b\right] = A\mathbb{E}\left[\mathbf{Y}\right]+b\text{, }$$ $$\mathbb{E}\left[A\mathbf{Y}b\right] = A\mathbb{E}[\mathbf{Y}]b\text{,} $$ and $$\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}\text{,}$$ where $\text{Cov}(\mathbf{Y})$ denotes the (variance-)covariance matrix of $\mathbf{Y}$.

The book says that using $\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}$, we can show that $\text{Cov}\left(\mathbf{Y}\right)$ is nonnegative definite for any random vector $\mathbf{Y}$.

So we have for $v \in \mathbb{R}^n$, $$v^{\prime} \text{Cov}\left(\mathbf{Y}\right) v = \text{Cov}\left(v^{\prime}\mathbf{Y}\right)\text{.}$$ Just looking at the matrix dimensions, I know that $v^{\prime}\mathbf{Y}$ is a $1 \times 1$ summation of terms. In particular, $$v^{\prime}\mathbf{Y} = \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}\begin{bmatrix} y_{1} \\ y_2 \\ \vdots \\ y_n\end{bmatrix} = \sum\limits_{i=1}^{n}y_iv_i\text{.}$$ Just given my machinery above, I'm not sure how to find $\text{Cov}\left(\sum\limits_{i=1}^{n}y_iv_i\right)$ without it being very messy.

Note that I already asked a very similar question here, but I would like to know what the author of the book is thinking by suggesting using $\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}$.

  • 1
    $\begingroup$ I must be missing something. You're asking how to show that the covariance of any random vector is non-negative definite. What does this have to do with $A$ or $b$? They seem to be irrelevant. Moreover, the demonstration is immediate (and has appeared in other threads on this site): because for any $x\in\mathbb{R}^n$, $x^\prime\text{cov}(Y)x$ is the variance of a random vector $Yx$, it is non-negative, QED. Is there perhaps some confusion in the text concerning what you're trying to prove? $\endgroup$ – whuber Jun 1 '15 at 21:23
  • $\begingroup$ @whuber The text states $\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}$ implies that $\text{Cov}\left(\mathbf{Y}\right)$ is nonnegative definite. The variance of a random vector in this text has not yet been defined. The machinery I have been given so far is that $\mathbb{E}\left[A\mathbf{Y}+b\right] = A\mathbb{E}\left[\mathbf{Y}\right]+b$, $\mathbb{E}\left[A\mathbf{Y}b\right] = A\mathbb{E}[\mathbf{Y}]b$, and $\text{Cov}\left(A\mathbf{Y}+b\right) = A\text{Cov}\left(\mathbf{Y}\right)A^{\prime}$, as shown in my post. $\endgroup$ – Clarinetist Jun 1 '15 at 22:05
  • $\begingroup$ @whuber I also know that a symmetric matrix $A$ is nonnegative definite if and only if $A = QQ^{\prime}$ for some square matrix $Q$ and that I can decompose $A$ as $PD(\lambda_i)P^{\prime}$, where $P$ is orthogonal. $\endgroup$ – Clarinetist Jun 1 '15 at 22:06
  • 1
    $\begingroup$ Sorry, the thing I termed "random vector" is just a random variable. Since you have a definition of covariances you certainly have a definition of variances. $\endgroup$ – whuber Jun 1 '15 at 22:15
  • 2
    $\begingroup$ Clarinetist, you just have to follow whuber's original suggestion.cov of a scalar=variance>=0. And take A=v ie 1xn 'matrix' $\endgroup$ – seanv507 Jun 1 '15 at 22:33

A square matrix $C = [C_{i,j}]$ is said to be positive semi-definite if and only if $$\sum_i \sum_j a_ia_j C_{i,j} \geq 0 ~ \forall a_i, a_j \in \mathbb R. \tag{1}$$ It is called positive definite if the inequality in $(1)$ is a strict inequality.

Now choose $A$ to be the row vector (a.k.a. $1\times n$ matrix) $[a_1, a_2, \ldots, a_n]$ so that $A\mathbf Y$ is the univariate random variable $Z= \sum_i a_iY_i$ and so $$\operatorname{cov}\left(A\mathbf Y+b\right) = \operatorname{var}(Z) \geq 0.\tag{2}$$ But, we are given that $$\operatorname{cov}\left(A\mathbf{Y}+b\right) = A\operatorname{cov}\left(\mathbf{Y}\right)A^{T}\tag{3}$$ where, for our chosen $A$, the right side of $(3)$ is just $\sum_i \sum_j a_i a_j \operatorname{cov}\left(\mathbf Y\right)_{i,j}$. We conclude from $(2)$ and $(3)$ that $$\sum_i \sum_j a_i a_j \operatorname{cov}\left(\mathbf Y\right)_{i,j}\geq 0~ \forall a_i, a_j \in \mathbb R,\tag{4}$$ that is, $\operatorname{cov}\left(\mathbf Y\right)$ is a positive semi-definite matrix.

Adapted from this previous answer of mine.

  • $\begingroup$ It took me a while to realize that $\text{Cov}\left(A\mathbf{Y}+b\right) = \text{Var}(Z)$, but it made sense once I realized that the covariance matrix of a vector in $\mathbb{R}^1$ is just the variance of the random variable in the vector. Thank you. By the way, $b$ isn't necessary for this proof at all, is it? $\endgroup$ – Clarinetist Jun 2 '15 at 4:05
  • $\begingroup$ No, $b$ was not needed except that we did use the special case $\operatorname{var}(Z+b)=\operatorname{var}(Z)$ without saying so in arriving at $(2)$. $\endgroup$ – Dilip Sarwate Jun 2 '15 at 4:20

In the result, plug $A = U'$, where $U$ is the matrix of eigenvectors. Writing the eigendecomposition $S = U \Lambda U^{T}$ reveals $Cov(UY)$ =$\Lambda$ which is a diagonal matrix

Now, the eigenvalues of a diagonal matrix are the diagonal entries. But, the diagonal entries are the variances and hence non-negative (this result can also be proved trivially) and therefore, the covariance matrix of $UY$ is positive definite.

Now use $Y_0 =UY$ and $A_{0} = U^{T}$ in the equation to get your result


Recall any valid covariance matrix is nonneg def. So for any random vector $Y$ if $\mbox{Cov}(Y)$ is defined, it has $\mbox{Cov}(Y) \geq 0$. Furthermore, another property of NND matrices is that, for any real vector $A$, and any NND matrix $\Sigma$, $A^\prime \Sigma A$ is also NND.

  • $\begingroup$ Unfortunately your sentence starting with "recall" isn't familiar to me. How would one prove this? $\endgroup$ – Clarinetist Jun 1 '15 at 20:07
  • $\begingroup$ @Clarinetist that would have been the ideal starting point for such a question as yours. If going a more axiomatic route, the full proof of my "recall" claim is in Thm 3.1 of these lecture notes. The rough sketch is using the singular value decomposition of the covariance matrix. $\endgroup$ – AdamO Jun 1 '15 at 20:16
  • $\begingroup$ I am looking at Theorem 3.1 right now - it looks like it is dependent on $\Sigma = AA^{\prime}$ for jointly normal distributions. The factorization $AA^{\prime}$ I know is a necessary and sufficient condition for nonnegative definite matrices, but is there a way to prove that an arbitrary covariance matrix can be factored as such (i.e., not just for jointly normal distributions)? $\endgroup$ – Clarinetist Jun 1 '15 at 20:22
  • $\begingroup$ @Clarinetist they're all twoway implications. All the nonprobabilistic claims should have been covered in a basic linear algebra class. If you haven't covered them, you would do well to study this more closely. For instance, none of the assumptions in Thm 3.1 involve jointly normal RVs. An arbitrary cov mx is symmetric (verify from def), so it's SVD gives rise to AA' as a factorization. $\endgroup$ – AdamO Jun 1 '15 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.