6
$\begingroup$

Are the sample mean and sample variance of correlated normal observations independent?

The classic theory relies on the independence of observations, but this is not the case. So as an example I tried to simulate 100k samples of 5 elements taken from a stationary, ergodic, normal random process with known mean and covariance, and I evaluated the cross distribution between the sample mean and the sample variance and I got this:sample mean vs sample std dev

So from this (and others) simulations I'm starting to believe that sample mean and sample variance for a normal process are independent even if the samples are correlated, but I don't know how to prove this. Any help?

$\endgroup$
  • $\begingroup$ You start out asking about "independence" and wind up writing about "correlation" as if they were equivalent--but they are not. Which of those are you trying to ask about? $\endgroup$ – whuber Jan 25 '16 at 23:02
  • $\begingroup$ independence of sample mean and of sample variance. I'll edit the question $\endgroup$ – xanz Jan 25 '16 at 23:23
6
$\begingroup$

The sample mean of a multivariate normal vector $\mathbf{X}=(X_1, X_2, \ldots, X_n)$ is a function of

$$M = X_1+X_2+\cdots X_n$$

and the sample variance is a function of the residual vector with components

$$Z_i = -X_1 - X_2 - \cdots - X_{i-1} + (n-1)X_i - X_{i+1} - \cdots - X_n,$$

$i=1, 2, \ldots, n$.

Let $\Sigma$ be the covariance matrix of $\mathbf{X}$. Write $\sigma_i$ for the sum of column (or row) $i$ of $\Sigma$, $\sigma_i = \Sigma_{1i} + \Sigma_{2i} + \cdots + \Sigma_{ni}$, and let $\sigma$ be the sum of all the entries of $\Sigma$. We may compute

$$\operatorname{Cov}(M, Z_i) = n\sigma_i - \sigma.$$

Because both $M$ and $Z_i$ are linear combinations of multivariate Normal variables, they are jointly Normal, whence they are independent if and only if their covariance is zero. Consequently $M$ is independent of all the $Z_i$ if and only if

$$n\sigma_1 = n\sigma_2 = \cdots = n\sigma_n = \sigma.$$

In other words, equality of the column sums guarantees independence of the mean and the components of the sample variance, whence it will guarantee independence of the mean and the sample variance itself.

Although the converse is not true--it is possible for $M$ not to be independent of the $Z_i$, yet for $M$ to be independent of the sample mean--this requires exceptional circumstances. In almost all cases, inequality of the column sums creates a dependence between the sample mean and sample standard deviation.

By definition, in a stationary process the covariances $\Sigma_{ij}$ may depend only on $i-j$. Although this does not guarantee the column sums are all equal, for large $n$ and a covariance that decays sufficiently rapidly with $|i-j|$, it will approximately be true, because in the limit the column sums are all equal:

$$\sigma_i = \sum_{j=-\infty}^\infty \Sigma_{ji} =\sum_{j=-\infty}^\infty \Sigma_{jk} =\sigma_k.$$

All that is required is the convergence of these sums.


A good way to see the dependence in the scatterplot is to render the points more carefully. When they are made semitransparent, you can see the underlying density better. A lowess smooth helps demonstrate a variation in the standard deviation with the mean in this example where $n=8$ and the column sums of $\Sigma$ vary appreciably.

Scatterplot

Here is the R code that generated it.

library(MASS)   # mvrnorm()
set.seed(17)
n <- 5e4        # Simulation size
d <- 8          # Dimension
k <- 4          # Size of upper block of Sigma
rho <- 0.99     # Correlation in upper block
mu <- rep(0, d) # Mean
Sigma <- outer(1:d, 1:d, function(i,j) ifelse(i <= k & j <= k, rho^abs(i-j), i==j))
colSums(Sigma)

x <- mvrnorm(n, mu, Sigma)
sim <- t(apply(x, 1, function(y) c(mean(y), sd(y))))

plot(sim, pch=16, cex=0.5, col="#00000008",
     xlab="Mean", ylab="SD")
i <- order(sim[, 1])
lines(sim[i, 1], lowess(sim[i, 2], f=1/20)$y, col="Red", lwd=2)
# g <- cut(sim[, 1], quantile(sim[, 1], seq(0, 1, by=0.025)))
# boxplot(sim[, 2] ~ g)
$\endgroup$
  • $\begingroup$ Wow. I only wish I knew statistics as good as you Dr. Whuber. Unfortunately it is not my field of studies, even if I often rely on it . Thank you for the thorough answer $\endgroup$ – xanz Jan 26 '16 at 16:16
  • 1
    $\begingroup$ Also, even if now it's trivial to see, I think is useful for future readers to note that if $n=2$ and if the two r.v. share the same variance then the sum of columns is automatically equal, and in that case the sample mean and variance are always independent even if the two samples are correlated $\endgroup$ – xanz Jan 26 '16 at 19:45
  • $\begingroup$ That is an excellent observation. When I started writing this answer I wanted to provide some intuition concerning why there could be lack of independence, so I analyzed the situation where $n=2$--and it was no good! $\endgroup$ – whuber Jan 26 '16 at 19:48
  • $\begingroup$ Could you please clarify why $Z_{i}$ is equivalent to a component of the sample variance? $\endgroup$ – half-pass Jul 26 '16 at 23:20
1
$\begingroup$

Only if $\Sigma$ is diagonal it appears. (see edit). I'm going to look at the covariance between $[\bar{X}, \ldots, \bar{X}]'$ and $X - [\bar{X}, \ldots, \bar{X}]'$. It's easier to work with the unsquared deviates rather than the sum of squared deviates.

Let $X \sim \text{Normal}(\mu, \Sigma)$, where $X, \mu \in R^n$. Let $\Sigma$ look like whatever you want it to. Not diagonal, because you don't want independence between samples. Just as long as it's symmetric and positive definite (I'm assuming we're dealing with full rank normals here).

Then I use bilinearity of $Cov(\cdot, \cdot)$.

$Cov(1\frac{1}{n}1'X, (I-\frac{1}{n}11')X) = \frac{1}{n}11' \Sigma (I-\frac{1}{n}11') = \frac{1}{n}11' \Sigma-\frac{1}{n}11' \Sigma \frac{1}{n}11' = \frac{1}{n}11' \Sigma-\frac{1}{n^2}11' \Sigma 11'$

As whuber points out, 0 covariance/correlation only implies independence in the case of normal vectors. This can be shown by writing down the density and seeing it factors.

Edit: whuber is correct that my final conclusion is incorrect. He came up with a more general criterion that guarantees independence (+1). Below I continue on verifying his answer with my notation. Let $e_i = (0,\ldots, 1,\ldots, 0)'$ be the vector with a one in the $i$th spot. Whuber's condition that the row or column sums be equal is equivalent to assuming that $\Sigma e_i = \Sigma e_k$ for $i \neq k$ or $e_i' \Sigma = e_k' \Sigma$. If we re-write $\Sigma 1$ as $ \sum_{i=l}^n \Sigma e_l$ then $\Sigma 1 = n \Sigma e_1$ and $Cov(1\frac{1}{n}1'X, (I-\frac{1}{n}11')X)$ becomes diagonal using the derivation above. Side note: apologies for overloading $\Sigma$.

$\endgroup$
  • 2
    $\begingroup$ Could you please explain how this demonstrates independence? After all, we all know that zero correlation is a necessary but not a sufficient condition for independence unless the two variables involved are jointly normal--but these are not. $\endgroup$ – whuber Jan 25 '16 at 23:01
  • $\begingroup$ Are you saying that if the samples are correlated, i.e. $\sum$ is not diagonal, there is no chance of the independence of mean and variance? $\endgroup$ – xanz Jan 26 '16 at 14:28
  • $\begingroup$ That conclusion is not correct. I'll post an analysis. $\endgroup$ – whuber Jan 26 '16 at 14:43
  • $\begingroup$ xanz, under normality. whuber, okey dokey I'm all ears. $\endgroup$ – Taylor Jan 26 '16 at 16:49
  • $\begingroup$ Sorry; I wasn't saying your conclusion was incorrect, Taylor. I was referring only to the preceding comment by xanz. Your previous analysis was incomplete, though, so I appreciate your efforts at fleshing it out. $\endgroup$ – whuber Jan 26 '16 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.