3
$\begingroup$

I appreciate the help I have been getting on this site today. I had help on a proof that $\text{Cov}\left(\mathbf{Y}\right)$ is nonnegative definite for any random vector $\mathbf{Y}$. According to the textbook I have, it follows from this statement that $\mathbf{Y}$ is nonsingular if and only if $\text{Cov}\left(\mathbf{Y}\right)$ is positive definite.

Now this doesn't make sense to me, since $\mathbf{Y} \in \mathbb{R}^n$, and I thought that only square matrices were nonsingular? Also, I know that $\text{Cov}\left(\mathbf{Y}\right)$ is positive definite if and only if there exists a nonsingular (square matrix) $Q$ such that $$\text{Cov}\left(\mathbf{Y}\right) = QQ^{\prime}\text{.}$$

So are they perhaps actually saying that $\text{Cov}\left(\mathbf{Y}\right) = \mathbf{Y}\mathbf{Y}^{\prime}$, where $\mathbf{Y}$ is a nonsingular square matrix rather than vector?

$\endgroup$
8
  • $\begingroup$ Do you mean $Y \in \mathbb{R}^n$, i.e. $Y$ is a random vector? $\endgroup$ Jun 2, 2015 at 5:21
  • $\begingroup$ @MatthewDrury Oh yes, will edit $\endgroup$ Jun 2, 2015 at 5:24
  • 1
    $\begingroup$ Given the focus of the book you are reading on linear models, I am wondering whether you have revealed all the assumptions it is making about $\mathbf{Y}$. Is this truly an arbitrary random vector, or is it perhaps the dependent variable in a linear model? And if so, what additional assumptions might have been made about it? $\endgroup$
    – whuber
    Jun 2, 2015 at 18:48
  • 2
    $\begingroup$ @whuber It looks like I must have skimmed over this page too much because there is one sentence I haven't seen before that basically clarifies everything: A random vector is referred to as singular or nonsingular depending on whether its covariance matrix is singular or nonsingular. Ugh. Apologies and thank you for your efforts. $\endgroup$ Jun 2, 2015 at 19:15
  • 1
    $\begingroup$ When your last comment appeared I was just in the process of asking whether this text defines "singular" in an unexpected way! (@Dougal guessed as much in a comment two hours ago.) That sentence resolves everything. $\endgroup$
    – whuber
    Jun 2, 2015 at 19:16

2 Answers 2

4
$\begingroup$

Your textbook may be referring to the distribution of $Y$ being a singular distribution: a distribution which doesn't have any point masses, but still its whole probability mass lies in a region of measure zero. This is kind of like $Y$ being in a lower-dimensional subset of $\mathbb R ^n$.

This is what I'd expect the statement to mean, but (as we'll see in a moment) this isn't an if and only if, just an only if. There may be some other conditions going on, or you may have misread it?

Let's call $\mathrm{Cov} Y$ just $\Sigma$ for brevity, and let $\mu = \mathbb E Y$.

First, if $\Sigma$ is singular, the distribution of $Y$ is singular.

As an example, suppose that $Y_1 \sim \mathcal{N}(0, 1)$ but $Y_2$ is always $0$. Then $\Sigma = \begin{bmatrix}1 & 0 \\ 0 & 0\end{bmatrix}$, a singular matrix, and $Y$ is concentrated on the $x$ axis of the plane (which is of measure zero).

In general, $\Sigma$ being singular means that there's some nonzero $q \in \mathbb R ^n$ with $\Sigma q = 0$. But note then that $\mathrm{Var}[ Y^T q] = q^T \Sigma q - (\mu^T q)^2 = -(\mu^T q)^2$, and since both variances and squared numbers must be nonnegative, we have both that $Y^T q$ has zero variance and $\mu^T q = 0$. Thus $Y$ lies only in the hyperplane $\{ y \mid q^T y = 0\}$, an $(n-1)$-dimensional subset of $\mathbb R ^n$.

This should make it clear that it's possible to have a singular distribution with a nonsingular covariance matrix, if $Y$ lies on any measure-zero subset other than a hyperplane through the origin.

For example, consider $Y$ to be uniformly distributed over the unit circle in $\mathbb{R}^2$; this has measure zero, but $$\mathrm{Cov}(Y_1, Y_2) = \mathbb{E}[Y_1 Y_2] = \mathbb{E}_{Y_1}\left[ \mathbb{E}\left[ Y_2 \mid Y_1 \right] \right] = \mathbb{E}_{Y_1}\left[ 0 \right] = 0,$$ and clearly $\mathrm{Var}(Y_1) = \mathrm{Var}(Y_2) = v > 0$ for some $v$, so $$\mathrm{Cov}[Y] = \begin{bmatrix}v & 0 \\ 0 & v\end{bmatrix}$$ which is nonsingular.

$\endgroup$
3
  • $\begingroup$ Thank you for the explanation. There are no other conditions imposed on the statement in my post other than $\mathbf{Y} \in \mathbb{R}^n$ being a random vector. ("Singular" in this textbook other than the matrix non-invertibility definition has not been defined.) The author even goes on to say that "By definition, $\mathbf{Y}$ nonsingular means precisely that $\text{Cov}\left(\mathbf{Y}\right)$ is positive definite," and I have always been told that definitions are iff statements. (and of course, this "by definition" statement isn't justified by any definition that has appeared so far.) $\endgroup$ Jun 2, 2015 at 15:07
  • 1
    $\begingroup$ @Clarinetist Yep, that sure sounds like an if and only if. I suppose the author is giving here a nonstandard definition of a singular distribution. What book is this? $\endgroup$
    – Danica
    Jun 2, 2015 at 17:17
  • $\begingroup$ This is Plane Answers to Complex Questions, in chapter 1. Is it possible that the author is just flat out wrong? $\endgroup$ Jun 2, 2015 at 17:34
0
$\begingroup$

Matrices can be singular/non singular. Singular matrix basically means that the matrix is not full rank. For a $N \times P$ matrix with $N>P$, the matrix is non-singular if the rank of the matrix is $P$. By this notion I guess you can say vectors are never singular but that is a rather peculiar way of saying it.

In terms of singular values, the result means you have $P$ non-zero singular values. Rank of ${\bf{YY}}'$ is equal to the number of non-zero singular values of ${\bf{Y}}$

When ${\bf{Y}}$ is a vector, ${\bf{YY}}'$ is always rank 1. If ${\bf{Y}}$ is a 2 $\times$ N on the other hand, based on the rank of ${\bf{Y}}$ you can tell if the covariance matrix is singular or not

$\endgroup$
9
  • $\begingroup$ It isn't at all clear how this post addresses the question, which concerns the sense in which a vector-valued random variable could be termed "singular." Could you perhaps edit it to make the connection evident? $\endgroup$
    – whuber
    Jun 2, 2015 at 13:08
  • $\begingroup$ The statement suggests $Y$ is a vector. Vectors are never singular in my opinion. They are 1-D objects in 1-D spaces. Hence, I discuss the result for the general $N \times P$ case. When $P = 1$, we get a vector $\endgroup$
    – Sid
    Jun 2, 2015 at 15:39
  • $\begingroup$ Dougal's answer suggests this interpretation may be misleading. $\endgroup$
    – whuber
    Jun 2, 2015 at 15:41
  • $\begingroup$ The question says (if and only if bit) that Cov(${\bf{Y}}$) is positive definite. Thus, it is not singular. If ${\bf{Y}}$ is a $N \times 1$ vector, ${\bf{YY}}'$ is always rank 1 and therefore singular. Dougal's answer considers the possible explanation that the covariance matrix is singular which though not the case in question, may be hinting at some point the author missed $\endgroup$
    – Sid
    Jun 2, 2015 at 16:05
  • 1
    $\begingroup$ Thank you. I am puzzled, though. Since $\mathbf{Y}$ is a random vector (or random matrix as your "$2\times N$" remark suggests), $\mathbf{YY^\prime}$ is a random matrix, whence its rank is a random variable, too. Whether or not it indicates non-singularity will depend on the probability distribution. And when $\mathbf Y$ is a random matrix, exactly what do you mean by its covariance matrix? That is ambiguous, but ordinarily it would just be the covariance of the components. However, its singularity--or lack thereof--has no apparent connection to the rank of $\mathbf{Y}$ itself. $\endgroup$
    – whuber
    Jun 2, 2015 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.