17
$\begingroup$

I am a noob in statistics, so could you guys please help me out here.

My question is the following: What does pooled variance actually mean?

When I look for a formula for pooled variance in the internet, I find a lot of literature using the following formula (for example, here: http://math.tntech.edu/ISR/Mathematical_Statistics/Introduction_to_Statistical_Tests/thispage/newnode19.html):

\begin{equation} \label{eq:stupidpooledvar} \displaystyle S^2_p = \frac{S_1^2 (n_1-1) + S_2^2 (n_2-1)}{n_1 + n_2 - 2} \end{equation}

But what does it actually calculate? Because when I use this formula to calculate my pooled variance, it gives me wrong answer.

For example, consider these "parent sample":

\begin{equation} \label{eq:parentsample} 2,2,2,2,2,8,8,8,8,8 \end{equation}

The variance of this parent sample is $S^2_p=10$, and its mean is $\bar{x}_p=5$.

Now, suppose I split this parent sample into two sub-samples:

  1. The first sub-sample is 2,2,2,2,2 with mean $\bar{x}_1=2$ and variance $S^2_1=0$.
  2. The second sub-sample is 8,8,8,8,8 with mean $\bar{x}_2=8$ and variance $S^2_2=0$.

Now, clearly, using the above formula to calculate the pooled/parent variance of these two sub-samples will produce zero, because $S_1=0$ and $S_2=0$. So what does this formula actually calculate?

On the other hand, after some lengthy derivation, I found the formula which produces the correct pooled/parent variance is:

\begin{equation} \label{eq:smartpooledvar} \displaystyle S^2_p = \frac{S_1^2 (n_1-1) + n_1 d_1^2 + S_2^2 (n_2-1) + n_2 d_2^2} {n_1 + n_2 - 1} \end{equation}

In the above formula, $d_1=\bar{x_1}-\bar{x}_p$ and $d_2=\bar{x_2}-\bar{x}_p$.

I found a similar formula with mine, for example here: http://www.emathzone.com/tutorials/basic-statistics/combined-variance.html and also in Wikipedia. Although I have to admit that they don't look exactly the same like mine.

So again, what does pooled variance actually mean? Shouldn't it mean the variance of parent sample from the two sub-samples? Or I am completely wrong here?

Thank you in advance.


EDIT 1: Someone says that my two sub-samples above are pathological since they have zero variance. Well, I could give you a different example. Consider this parent sample:

\begin{equation} \label{eq:parentsample2} 1,2,3,4,5,46,47,48,49,50 \end{equation}

The variance of this parent sample is $S^2_p=564.7$, and its mean is $\bar{x}_p=25.5$.

Now, suppose I split this parent sample into two sub-samples:

  1. The first sub-sample is 1,2,3,4,5 with mean $\bar{x}_1=3$ and variance $S^2_1=2.5$.
  2. The second sub-sample is 46,47,48,49,50 with mean $\bar{x}_2=48$ and variance $S^2_2=2.5$.

Now, if you use "literature's formula" to compute the pooled variance, you will get 2.5, which is completely wrong, because the parent/pooled variance should be 564.7. Instead, if you use "my formula", you will get correct answer.

Please understand, I use extreme examples here to show people that the formula indeed wrong. If I use "normal data" which doesn't have a lot of variations (extreme cases), then the results from those two formulae will be very similar, and people could dismiss the difference due to rounding error, not because the formula itself is wrong.

$\endgroup$
  • $\begingroup$ Some related links to help: stats.stackexchange.com/q/214834/3277, stats.stackexchange.com/q/12330/3277, stats.stackexchange.com/q/43159/3277. $\endgroup$ – ttnphns Sep 12 '17 at 14:17
  • $\begingroup$ As a statistics student, I think the first fomula is used to estimate variance of difference of two sample mean, so that you can form Z- statistics ~ N(0,1) under Null Hypothesis. However, the second formula is the variance of concatenation of two samples. If we want to distincguish them, no only from literal aspect but also from their functionality. $\endgroup$ – Travis Aug 23 at 12:13
14
$\begingroup$

Put simply, the pooled variance is an (unbiased) estimate of the variance within each sample, under the assumption/constraint that those variances are equal.

This is explained, motivated, and analyzed in some detail in the Wikipedia entry for pooled variance.

It does not estimate the variance of a new "meta-sample" formed by concatenating the two individual samples, like you supposed. As you have already discovered, estimating that requires a completely different formula.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The assumption of "equality" (that is, same population realized those samples) is not necessary in general to define what it is - "pooled". Pooled simply means averaged , omnibus (see my comment to Tim). $\endgroup$ – ttnphns Sep 13 '17 at 3:50
  • $\begingroup$ @ttnphns I think the equality assumption is necessary for giving the pooled variance a conceptual meaning (which the OP asked for) that goes beyond just verbally describing the mathematical operation it performs on the sample variances. If the population variances are not assumed equal, then it's unclear what we could consider the pooled variance to be an estimate of. Of course, we could just think about it as being an amalgamation of the two variances and leave it at that, but that's hardly enlightening in the absence of any motivation for wanting to combine the variances in the first place. $\endgroup$ – Jake Westfall Sep 13 '17 at 4:41
  • $\begingroup$ Jake, I'm not in disagreement with that, given the specific question of the OP, but I wanted to speak about definition of the word "pooled", that's why I said, "in general". $\endgroup$ – ttnphns Sep 13 '17 at 4:46
  • $\begingroup$ @JakeWestfall Your answer is the best answer so far. Thank you. Although I am still not clear about one thing. According to Wikipedia, pooled variance is a method for estimating variance of several different populations when the mean of each population may be different, but one may assume that the variance of each population is the same. $\endgroup$ – Hanciong Sep 15 '17 at 2:58
  • $\begingroup$ @JakeWestfall: So if we are calculating pooled variance from two different populations with different means, what does it actually calculate? Because the first variance is measuring the variation with respect to the first mean, and the second variance is with respect to the second mean. I don't know what additional information can be gained from calculating it. $\endgroup$ – Hanciong Sep 15 '17 at 3:01
11
$\begingroup$

Pooled variance is used to combine together variances from different samples by taking their weighted average, to get the "overall" variance. The problem with your example is that it is a pathological case, since each of the sub-samples has variance equal to zero. Such pathological case has very little in common with the data we usually encounter, since there is always some variability and if there is no variability, we don't care about such variables since they carry no information. You need to notice that this is a very simple method and there are more complicated ways of estimating variance in hierarchical data structures that are not prone to such problems.

As about your example in the edit, it shows that it is important to clearly state your assumptions before starting the analysis. Let's say that you have $n$ data points in $k$ groups, we would denote it as $x_{1,1},x_{2,1},\dots,x_{n-1,k},x_{n,k}$, where the $i$-th index in $x_{i,j}$ stands for cases and $j$-th index stands for group indexes. There are several scenarios possible, you can assume that all the points come from the same distribution (for simplicity, let's assume normal distribution),

$$ x_{i,j} \sim \mathcal{N}(\mu, \sigma^2) \tag{1} $$

you can assume that each of the sub-samples has its own mean

$$ x_{i,j} \sim \mathcal{N}(\mu_j, \sigma^2) \tag{2} $$

or, its own variance

$$ x_{i,j} \sim \mathcal{N}(\mu, \sigma^2_j) \tag{3} $$

or, each of them have their own, distinct parameters

$$ x_{i,j} \sim \mathcal{N}(\mu_j, \sigma^2_j) \tag{4} $$

Depending on your assumptions, particular method may, or may not be adequate for analyzing the data.

In the first case, you wouldn't be interested in estimating the within-group variances, since you would assume that they all are the same. Nonetheless, if you aggregated the global variance from the group variances, you would get the same result as by using pooled variance since the definition of variance is

$$ \mathrm{Var}(X) = \frac{1}{n-1} \sum_i (x_i - \mu)^2 $$

and in pooled estimator you first multiply it by $n-1$, then add together, and finally divide by $n_1 + n_2 - 1$.

In the second case, means differ, but you have a common variance. This example is closest to your example in the edit. In this scenario, the pooled variance would correctly estimate the global variance, while if estimated variance on the whole dataset, you would obtain incorrect results, since you were not accounting for the fact that the groups have different means.

In the third case it doesn't make sense to estimate the "global" variance since you assume that each of the groups have its own variance. You may be still interested in obtaining the estimate for the whole population, but in such case both (a) calculating the individual variances per group, and (b) calculating the global variance from the whole dataset, can give you misleading results. If you are dealing with this kind of data, you should think of using more complicated model that accounts for the hierarchical nature of the data.

The fourth case is the most extreme and quite similar to the previous one. In this scenario, if you wanted to estimate the global mean and variance, you would need a different model and different set of assumptions. In such case, you would assume that your data is of hierarchical structure, and besides the within-group means and variances, there is a higher-level common variance, for example assuming the following model

$$ \begin{align} x_{i,j} &\sim \mathcal{N}(\mu_j, \sigma^2_j) \\ \mu_j &\sim \mathcal{N}(\mu_0, \sigma^2_0) \\ \sigma^2_j &\sim \mathcal{IG}(\alpha, \beta) \end{align} \tag{5} $$

where each sample has its own means and variances $\mu_j,\sigma^2_j$ that are themselves draws from common distributions. In such case, you would use a hierarchical model that takes into consideration both the lower-level and upper-level variability. To read more about this kind of models, you can check the Bayesian Data Analysis book by Gelman et al. and their eight schools example. This is however much more complicated model then the simple pooled variance estimator.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I have updated my question with different example. In this case, the answer from "literature's formula" is still wrong. I understand that we are usually dealing with "normal data" where there is no extreme case like my example above. However, as mathematicians, shouldn't you care about which formula is indeed correct, instead of which formula applies in "everyday/common problem"? If some formula is fundamentally wrong, it should be discarded, especially if there is another formula which holds in all cases, pathological or not. $\endgroup$ – Hanciong Sep 12 '17 at 11:51
  • $\begingroup$ Btw you said there are more complicated ways of estimating variance. Could you show me these ways? Thank you $\endgroup$ – Hanciong Sep 12 '17 at 11:57
  • 2
    $\begingroup$ Tim, pooled variance is not the total variance of the "combined sample". In statistics, "pooled" means weighted averaged (when we speak of averaged quantities such as variances, weights being the n's) or just summed (when we speak of sums such as scatters, sums-of-squares). Please, reconsider your terminology (choice of words) in the answer. $\endgroup$ – ttnphns Sep 12 '17 at 13:32
  • 1
    $\begingroup$ Albeit off the current topic, here is an interesting question about "common" variance concept. stats.stackexchange.com/q/208175/3277 $\endgroup$ – ttnphns Sep 12 '17 at 13:51
  • 1
    $\begingroup$ Hanciong. I insist that "pooled" in general and even specifically "pooled variance" concept does not need, in general, any assumption such as: groups came from populations with equal variances. Pooling is simply blending (weighted averaging or summing). It is in ANOVA and similar circumstances that we do add that statistical assumption. $\endgroup$ – ttnphns Sep 15 '17 at 6:03
1
$\begingroup$

The problem is if you just concatenate the samples and estimate its variance you're assuming they're from the same distribution therefore have the same mean. But we are in general interested in several samples with different mean. Does this make sense?

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The use-case of pooled variance is when you have two samples from distributions that:

  • may have different means, but
  • which you expect to have an equal true variance.

An example of this is a situation where you measure the length of Alice's nose $n$ times for one sample, and measure the length of Bob's nose $m$ times for the second. These are likely to produce a bunch of different measurements on the scale of millimeters, because of measurement error. But you expect the variance in measurement error to be the same no matter which nose you measure.

In this case, taking the pooled variance would give you a better estimate of the variance in measurement error than taking the variance of one sample alone.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your answer, but I still don't understand about one thing. The first data gives you the variance with respect to Alice's nose length, and the second data gives you the variance with respect to Bob's nose length. If you are calculating a pooled variance from those data, what does it mean actually? Because the first variance is measuring the variation with respect to Alice's, and the second with respect to Bob's, so what additional information can we gained by calculating their pooled variance? They are completely different numbers. $\endgroup$ – Hanciong Sep 15 '17 at 3:08
0
$\begingroup$

Through pooled variance we are not trying to estimate the variance of a bigger sample, using smaller samples. Hence, the two examples you gave don't exactly refer to the question.

Pooled variance is required to get a better estimate of population variance, from two samples that have been randomly taken from that population and come up with different variance estimates.

Example, you are trying to gauge variance in the smoking habits of males in London. You sample two times, 300 males from London. You end up getting two variances (probably a bit different!). Now since, you did a fair random sampling (best to your capability! as true random sampling is almost impossible), you have all the rights to say that both the variances are true point estimates of population variance (London males in this case).

But how is that possible? i.e. two different point estimates!! Thus, we go ahead and find a common point estimate which is pooled variance. It is nothing but weighted average of two point estimates, where the weights are the degree of freedom associated with each sample.

Hope this clarifies.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Although I am very late to the conversation maybe I can add something helpful:
It seems to me that the OP wants to know why (what for) we would need a pooled variability estimate $\hat\sigma_{pooled}$ as a weighted average of two samples (be it variance or standard deviation).

As far as I am aware the main practical need for this kind of dispersion measure arises from wanting to compare means of (sub-)groups: so if I want to compare the average nose length for 1) people who did not undergo a gene therapy, 2) people who underwent gene therapy A and 3) people who underwent gene therapy B.
To be better able to compare the amount of the mean differences in length (mm) I divide the mean difference, say, $e=\bar x_{Control}-\bar x_{GTA}=30mm-28mm=2mm$ by the variability estimate (here standard deviation). Depending on the size of the square root of pooled variance (pooled standard deviation) we can better judge the size of the 2mm difference between those groups (e.g., $d=2mm/0.5mm=4$ vs. $d=2mm/4mm=0.5$ --> Does gene therapy A do something to the nose length? And if so, how much? When $d=4$ or $2 \pm 0.5mm$ there seems to be a "stable" or "consistent" or "big" (compared to the variability) difference between the mean nose lengths, when $d=0.5$ or $2 \pm 4mm$ it does not seem so much, relatively speaking. In case all values within both groups are the same and therefore there is no variability within the groups, $d$ would not be defined but the interpretation would be $2 \pm 0mm=2mm$ exactly).
This is the idea of effect size (first theoretically introduced by Neyman and Pearson as far as I know, but in one kind or another used well before, see Stigler, 1986, for example).
So what I am doing is comparing the mean difference between groups with the mean differences within those same groups, i.e weighted average of variances (standard deviations). This makes more sense than to compare the mean difference between (sub-)groups with the mean difference within the "whole" group, because, as you (Hanciong) have shown, the variance (and standard deviation) of the whole group contains the difference(s) of the group means as well.

The theoretical need for the measure arises from being able to use the $t$-distribution to find the probability for the observed mean difference or a more extreme one, given some expected value for the mean difference (p-value for e.g., Null-Hypothesis-Significance-Test, NHST, or Neyman-Pearson hypothesis test or Fisher hypothesis test, confidence intervals etc.): $p(e \ge e_{observed}|\mu_e=0)$.
As far as I know the p-value obtained by the $t$-distribution (and especially the $F$-distribution in cases with more than 2 means to compare) will give correct estimates for the probability only when both (or all) samples are drawn from populations with equal variances (homogeneity of variance, as pointed out in the other answers already; this should be described in (more) detail in most statistics textbooks). I think all distributions based on the normal distribution ($t$, $F$, $\chi^2$) assume a variance of more than 0 and less than $\infty$, so it would be impossible to find the p-value for a case with a within variability of 0 (in this case you would obviously not assume to have drawn your sample from a normal distribution).
(This also seems intuitively reasonable: if I want to compare two or more means then the precision of those means should be the same or at least comparable:
if I run my gene therapy A on people whose nose lengths are quite similar, say $\bar x \pm 0.5mm$ but have a group of people with high variability in nose lengths in my control group, say $\bar x \pm 4mm$ it doesn't seem fair to directly compare those means, because those means do not have the same "mean-meaning"; in fact the very much higher variance/standard deviation in my control group could be indicating further subgroups, maybe differences of nose lengths due to differences on some gene.)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.