What is the correlation if the standard deviation of one variable is 0?

As I understand, we can get correlation by normalizing covariance using the equation

$$\rho_{i,j}=\frac{cov(X_i, X_j)}{\sigma_i \sigma_j}$$

where $\sigma_i=\sqrt{E[(X_i-\mu_i)^2]}$ is the standard deviation of $X_i$.

My concern is what if the standard deviation equals zero? Is there any condition that guarantees it cannot be zero?

Thanks.

-
No variable that has standard deviation 0 could possibly be correlated with another (non-constant) variable. Correlation is a measure of how large/small values in one variable correspond to large/small values in another variable - if one of the variables is equal to a constant with probability 1 (a consequence of having standard deviation 0), then it can't possibly give information about whether the other variable is small or large. I don't know what the convention is but it seems like the correlation should be defined as 0 in that case. –  Macro Nov 14 '11 at 4:57
Thanks a lot Macro. I think your idea is the same as the answer below. However, I couldn't vote your comment up due to limitation in points. Thanks. –  chepukha Nov 14 '11 at 5:15
You have accepted an answer already, and so I will write just a comment. If a random variable $Y$ has standard deviation $\sigma_Y = 0$, then $$\text{cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]=0$$ for any other random variable $X$ (since $(Y-\mu_Y)=0$ with probability $1$). Thus, the definition of the correlation coefficient $\rho_{X,Y}=\frac{\text{cov}(X,Y)}{\sigma_X\sigma_Y}$ gives the indeterminate form $\frac{0}{0}$. It is conventional to define $\rho_{X,Y}$ to be equal to $0$ in this case, and this can be defended on the grounds of the limiting value of $\rho_{X,Y}$ as $\sigma_Y \to 0$ etc. –  Dilip Sarwate Nov 14 '11 at 12:00
@Dilip, if it is an answer it should go as an answer. It shouldn't matter whether an answer is accepted already. –  Andy W Nov 14 '11 at 13:10
@Dilip The problem with the $\frac{0}{0}$ form is that even if it can be made to have a definite value by means of a limiting operation, the value depends on how you take the limit. Whence, the argument that $\rho_{X,Y}= 0$ is incomplete (and unconvincing). Can you cite a source that adopts this convention and supports it with a valid reason? –  whuber Nov 14 '11 at 15:11

It's true that, if one of your SD's is 0, that equation is undefined. However, a better way to think about this is that if one of your SD's is 0, there is no correlation. In loose conceptual terms, a correlation is telling you about how one variable moves around as the other variable moves around. An SD of 0 implies that variable is not 'moving around'. You would have to have a vector of a constant, such as rep(constant, n_times).

-
Thanks a lot. I think that makes sense. It's interesting that I haven't seen any textbook mention that case. –  chepukha Nov 14 '11 at 5:14

The other thing to think about are the underlying assumptions when we talk about means and standard deviations, and correlations.

If we are talking about a data sample, one common assumption is that the data is (at least approximately) normally distributed, or can be transformed such that it is (e.g. via a log transform). If you observe a standard deviation of zero, there are two scenarios: either the standard deviation is in fact nonzero, but very small, and therefore the dataset you have has samples that are all on the mean value (this could, for example, happen if you are measuring data at a coarse level of precision); or the model is misspecified.

In this second scenario, the standard deviation, and consequently the correlation, is a meaningless measure.

More generally, the underlying distributions must both have finite second moments, and therefore non-zero standard deviations, for the correlation to be a valid concept.

-
It may be worth noting that the original question is about (theoretical) distributions, not about data. –  whuber Nov 14 '11 at 15:12
If that is the case, then a standard deviation of zero would imply a degenerate distribution with measure only at the mean (i.e. the constant function) ... again the standard deviation only makes sense the underlying distribution is normal. If the standard deviation is zero, the PDF of the Gaussian is not properly defined, and hence not permissible in the model. –  tdc Nov 14 '11 at 15:34
I'm surprised at the appearance of Gaussians in your comment, Tom. This seems like an unnecessary restriction. Requiring existence of a pdf also seems restrictive (after all, no discrete distribution has a pdf). Note, too, that the SD is well defined--"meaningful"--whenever the second moment is finite, and this includes probability atoms (your "Dirac delta" functions). –  whuber Nov 14 '11 at 15:56
Ok I agree was probably being overly restrictive, but generally this is what people mean by the SD. e.g. from Wolfram: "Standard deviation can be defined for any distribution with finite first two moments, but it is most common to assume that the underlying distribution is normal." Do you take my point though, that if the SD = 0 for one of the variables, the basic assumptions underlying the statistical concept of correlation are not being met? –  tdc Nov 14 '11 at 17:10
Yes, Tom, your last statement is spot on and I accept it gladly. However, the idea it expresses does not appear very prominently in your reply; if it's there, it's buried in the remarks about normal distributions, logs, delta functions, and the focus on data rather than the distributions themselves. BTW, one should be careful about statistical statements appearing on the Wolfram site: it is so heavily oriented towards mathematics that its characterizations about statistical practice can be questionable. Here, it's dead wrong: the use of SD goes way beyond Normal-distribution settings. –  whuber Nov 14 '11 at 17:36