8
$\begingroup$

Let $X\sim\mathrm{Binomial}(n,p)$. We know that $\mathrm{E}[X]=np$ and $\mathrm{Var}[X]=np(1-p)$. Does this imply that the sample mean $\bar x$ and the sample variance $s^2$ are dependent of each other? Or does it just mean that the population variance can be written as a function of the population mean?

$\endgroup$
13
$\begingroup$

$\bar x$ and $s^2$ are random variables. We can work out their joint distribution. Let's try the simplest possible nontrivial case, that of a sample of size $2$ from a Binomial$(1,p)$ distribution. There are only four possibilities for that sample, which are hereby tabulated along with their probabilities (computed from the independence of the two sample elements):

First value | Second value | Mean | Variance | Probability
          0 |            0 |    0 |        0 | (1-p)^2
          0 |            1 |  1/2 |      1/2 | (1-p)p
          1 |            0 |  1/2 |      1/2 | p(1-p)
          1 |            1 |    1 |        0 | p^2

The mean perfectly predicts the variance in this example. Thus, provided all probabilities are nonzero (that is, $p$ is neither $0$ nor $1$), the sample mean and sample variance are not independent.

An interesting question is whether, if in a family of distributions the mean determines the variance, the sample mean and sample variance can be independent. The answer is yes: take any family of Normal distributions in which the variance depends on the mean such as the set of all Normal$(\mu, \mu^2)$ distributions. No matter which of these distributions governs the sample, the sample mean and sample variance will be independent, because that's the case for any Normal distribution.

This analysis suggests that questions about the structure of a family of distributions (which concern $n$, $p$, $\mu$, and so forth) have no bearing on questions of independence of statistics of samples from any given element of the family.

$\endgroup$
  • $\begingroup$ But maybe that's because the normal distribution is a "special" case? I mean, it is known that, for any normal distribution, it is true that the sample mean is independent of the sample variance. But what happens if we are dealing with a distribution that it's not a normal distribution? $\endgroup$ – user6874652 Dec 30 '17 at 21:31
  • 1
    $\begingroup$ Typically the sample mean and sample variance are not independent. It makes no difference what family of distributions the distribution might be part of. $\endgroup$ – whuber Dec 30 '17 at 22:22
  • $\begingroup$ @whuber : Except that with $N(\mu, \sigma^2)$ the sample mean and sample variance are independent. $\endgroup$ – Michael Hardy Dec 30 '17 at 23:28
  • $\begingroup$ @Michael Thanks. I already noted that in the body of the answer. $\endgroup$ – whuber Dec 30 '17 at 23:29
  • $\begingroup$ @whuber: thanks for the analysis. Could you also please disclose the R code? Many thanks. $\endgroup$ – Maximilian Jan 2 '18 at 11:37
8
$\begingroup$

The property that, for an i.i.d. sample, the sample mean and the sample variance are independent, is a characterization of the normal distribution: for no other distribution such a property holds.

See Patel, J. K., & Read, C. B. (1982). Handbook of the normal distribution, p. 81 in the 1st 1982 edition, in chapter "Characterizations" (may have changed pages in the 2nd edition of 1996).

So for any other distribution, the sample mean and the sample variance are statistically dependent.

The general result regarding the sample mean and the sample variance from an i.i.d. sample of any distribution that has moments up to the 3d, is the following (using the unbiased estimator for the variance):

$$\operatorname{Cov} (\bar X, s^2) = E(\bar X s^2) - E(x)\operatorname{Var}(x) = \frac 1n E[X-E(x)]^3$$

In words, the covariance between the sample mean and the sample variance is equal to the third central moment, divided by $n$. Consequences:

1) As sample size increases the two tend to become uncorrelated.

2) For any distribution that has the third central moment equal to zero, they are uncorrelated (although they remain dependent, for all distributions except the normal). This of course includes all distributions symmetric about their mean, but also other distributions that are non-symmetric about their mean but still, have the third central moment equal to zero, see this thread.

$\endgroup$
  • $\begingroup$ (+1) The hyperlink is dead for me. $\endgroup$ – COOLSerdash Dec 31 '17 at 8:44
  • $\begingroup$ @COOLSerdash It works for me. It links to an Amazon page, maybe that is blocked for you? $\endgroup$ – Graipher Dec 31 '17 at 12:50
  • $\begingroup$ @COOLSerdash Thanks. As mentioned, the hyperlink seems valid. Just search "Handbook of normal distribution Patel Read". $\endgroup$ – Alecos Papadopoulos Dec 31 '17 at 12:58
  • $\begingroup$ (+1) I suspected that this might be the case but have never seen a formal statement of this fact. Are there non-normal distributions for which the sample mean and sample variance are uncorrelated? $\endgroup$ – John Coleman Dec 31 '17 at 14:54
  • 1
    $\begingroup$ @AlecosPapadopoulos Yes, of course. If so then it would be an interesting example of when uncorrelated doesn't imply independent. I haven't worked out all of the details, but U(0,1) seems to work. $\endgroup$ – John Coleman Dec 31 '17 at 15:32
2
$\begingroup$

An extreme case is $\operatorname{Bernoulli}(p) = \operatorname{Binomial}(1,p).$ Consider a sample of size (capital) $N:$ \begin{align} Ns^2 = \sum_{k=1}^N (x_k - \overline x)^2 = {} & \left( \sum_k x_k^2 \right) - \left( 2\overline x \sum_i x_k \right) + \left( N\overline x^2 \right) \\[10pt] = {} & \left( \sum_k x_k \right) - 2\overline x \sum_k x_k + \left( n\overline x^2 \right) \\ & \text{since $x_k = 0$ or $1$, so $x_k^2=x_k$} \\[12pt] = {} & N\overline x - 2N\overline x^2 + N \overline x^2 \\[10pt] = {} & N \overline x(1-\overline x), \\[10pt] \text{so } s^2 = {} & \overline x(1-\overline x). \end{align} Thus when (lower-case) $n$ is $1,$ then the sample mean determines the sample variance, so they are far from independent. But the sample variance does not quite fully determine the sample mean, since there are two values of $\overline x$ that yield the same value of $\overline x(1-\overline x).$

When both $np$ and $n(1-p)$ are large, then I expect the sample mean and sample variance would be nearly independent since the distribution is nearly normal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.