7
$\begingroup$

I am trying to figure out the covariance between sample mean and sample variance from a population. We DO NOT know whether the population is normal (if it's normal, then the covariance is zero between sample mean and sample variance).

Here is my attempt:

$$Cov[\bar{y}, s^2] = \mathbb{E}[\bar{y}s^2]-\mathbb{E}[\bar{y}]\mathbb{E}[s^2]$$

I already know $\mathbb{E}[\bar{y}]$ and $\mathbb{E}[s^2]$ but can not figure out $\mathbb{E}[\bar{y}s^2]$.

Any help would be appreciated!

$\endgroup$
6
  • 4
    $\begingroup$ -1 for misrepresenting your original question (and thereby blaming three community members for your error): the entire editing record is available for inspection. $\endgroup$
    – whuber
    Aug 23, 2023 at 13:44
  • 1
    $\begingroup$ Consider that the original question title was "Covariance between sample mean and sample variance". The most seasoned users of CV definitely do not need to read the whole question to try to see if you were inconsistent, because they usually know how to answer this type of question with minimal effort (see whuber's great answer for example). I would advise OP to be more careful with their future questions. Also, this might change depending on which corners of the internet you use, but all-caps text is usually considered very rude. $\endgroup$
    – Firebug
    Aug 23, 2023 at 14:49
  • 1
    $\begingroup$ So, for all these reasons, I rolled-back your question to match the original. Consider asking a new question with the correct title. $\endgroup$
    – Firebug
    Aug 23, 2023 at 14:50
  • 1
    $\begingroup$ Admittedly, the OP's original question needed clarification: Although the title & opening sentence both said "variance", the offered equation indicated "standard deviation" was perhaps intended. But to be fair (especially to a New contributor), wouldn't the appropriate response in such case be to post a comment asking the OP to clarify, rather than to do what was done (i.e. someone other than the OP wrote words identified as the OP's). The OP said "Here is my attempt: [...]" and "I already know [...]". It seems totally inappropriate for anyone other than the OP to fill in those ellipses. $\endgroup$
    – r.e.s.
    Aug 23, 2023 at 15:37
  • 1
    $\begingroup$ @r.e.s. I see your point. I believe people are much more likely to make mistakes with notation than with words. If you use words to describe a problem, I'm more inclined to believe that was your intention if it conflicts with notation you introduce later. In a community where a good portion of questions go unanswered, trying to improve the chances of a question is a good thing. $\endgroup$
    – Firebug
    Aug 23, 2023 at 15:45

1 Answer 1

11
$\begingroup$

Consider a sample of size $n$ with values $(y_1, y_2, \ldots, y_n)$ drawn independently and randomly. The sample mean is

$$\bar y = \frac{1}{n}\sum_i y_i$$

and the sample variance is

$$s^2_y = \frac{1}{n-1} \left(\sum_i y_i^2 - \frac{1}{n}\left[\sum_j y_j\right]^2\right)$$

where all summation indexes range from $1$ through $n.$

Consequently, $\bar y s^2_y$ is a homogeneous cubic form in the $y_i.$ This means it is a linear combination of terms of the form $y_i^3,$ $y_iy_j^2,$ and $y_iy_jy_k$ where $i\ne j\ne k.$ Because the $y_i$ are identically distributed, the actual index values don't matter: we merely have to count how many of each kind of term there is and multiply their (common) coefficients by their counts.

Write $\mu_k$ for the raw $k^\text{th}$ moment $\mu_k = E[Y^k]$ where $Y$ is any random variable with this common distribution. Plug these formulas for $\bar y$ and $s^2_y$ into the expectations to find the coefficients and do the counting to obtain

$$E[\bar y s^2_y] = \frac{1}{n}(\mu_3 - (n-2)\mu_1^3 + (n-3)\mu_1\mu_2)$$

and

$$E[\bar y] = \mu_1;\ E[s^2_y] = \mu_2 - \mu_1^2$$

(as is well known).

For example, the coefficient of $y_1^3$ in $\bar y s^2_y$ is $1/n$ (from the coefficient of $y_1$ in $\bar y$) multiplied by $1/(n-1)$ times $1 - 1/n$ (from the coefficient of $y_1^2$ in $s^2_y$). Because there are $n$ such terms $y_1^3, y_2^3, \ldots, y_n^3,$ the coefficient of $\mu_3$ must be

$$n\left(\frac{1}{n}\right)\left(\frac{1}{n-1}\right) \left(1 - \frac{1}{n}\right) = \frac{1}{n}.$$

The other two coefficients in $E[\bar y s^2_y]$ are found similarly.

Thus, applying a tiny bit more algebra, we find

$$\operatorname{Cov}(\bar y, s^2_y) = E[\bar y s^2_y] - E[\bar y]E[s^2_y] = \frac{1}{n}(\mu_3 + 2\mu_1^3 - 3 \mu_1\mu_2).$$

In particular, in any symmetric distribution $\mu_1=\mu_3=0$ and the covariance must then be zero (but this doesn't imply $\bar y$ and $s^2_y$ are independent unless the distribution is Normal).


It is so easy to make little algebraic mistakes that an independent check is worthwhile. Here is R code to draw repeated samples from any (finite) population and compute their means and variances. At the end it prints the covariance of the means and the variances followed by the value given by this formula. Here is an example:

    Sample    Formula 
0.06382658 0.06380878 

They agree closely, with small differences attributed to the randomness of the simulation.

#
# Create any finite population.
#
pop <- rexp(30)
#
# Create a sampling distribution of sample means and sample variances.
#
n <- 10 # Sample size: must be 2 or greater
sim <- replicate(5e4, {
  y <- sample(pop, n, replace = TRUE)
  c(mean(y), var(y))
})
#
# Compare the covariance of the simulation to the formula.
#
mu <- function(x, k = 1) mean(x^k) # Raw moments of `x`
v <- (mu(pop, 3) + 2 * mu(pop, 1)^3 - 3 * mu(pop, 1) * mu(pop, 2)) / n
c(Sample = cov(sim[1, ], sim[2, ]), Formula = v)
$\endgroup$
9
  • 2
    $\begingroup$ I was wondering whether you meant SD instead of variance and so I studied the record of edits of your question before I answered. The original question, written by you, repeatedly asks about the "sample variance" -- so please don't blame the editors for altering your question. In general all we can hope for is to approximate the covariance between the sample mean and sample SD, perhaps using the delta method. If you still want to ask that question, then I suggest creating a new thread (perhaps referencing this one to avoid any confusion). $\endgroup$
    – whuber
    Aug 23, 2023 at 13:41
  • 1
    $\begingroup$ @JingyangZhang See the original question $\endgroup$
    – Firebug
    Aug 23, 2023 at 14:52
  • 1
    $\begingroup$ @JingyangZhang Hi Jingyang, I added Mathjax to your post and also added the squares (since you mentioned sample variance in the title and multiple times in the body I assumed it was an oversight in terms of notation). However, note that my edit was approved 13 hours ago and whuber answered your question 19 hours ago. So whuber was indeed answering your original question. $\endgroup$ Aug 23, 2023 at 15:08
  • 4
    $\begingroup$ Hi Whuber, Firebug, and Matenmakkers, I would like to sincerely apologize for all the confusion. It seems like I did put sample variance in my original post, while what I meant was sample standard deviation. $\endgroup$ Aug 23, 2023 at 15:23
  • 2
    $\begingroup$ That kind of thing happens to many (if not all) of us, where we mean one thing but write another. Thank you for your apology. $\endgroup$
    – whuber
    Aug 23, 2023 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.