When calculating sample variance around a sample mean can I take degrees of freedom to be a fraction

Question

Normally if you have $N$ samples of a random variable, from which you estimate both the sample mean and sample variance, you need to account for the missing degree of freedom in the sample variance:

$$Var(X) = \frac{1}{N-1} \sum^N_{i=1} (x_i - \bar{x})^2 $$

But what if the sample mean is estimated from a larger dataset that includes the $N$ samples, as well as some others (say you are interested in a timeseries of variance over some time windows, while estimating the mean over the entire time range). Say the larger dataset used for the mean has $(N + M)$ points, would it be sensible to divide by $N - \frac{N}{N + M}$?

Strictly speaking, there are (at least) $N$ degrees of freedom in the calculation for variance, but perhaps the above would be less biased.

Answer 1

You are considering the sample mean:

$$\bar{X}_{N+M} = \frac{1}{N+M} \sum_{i=1}^{N+M} X_i.$$

The sum-of-squared deviations around this value is:

$$\begin{align} \sum_{i=1}^N (X_i - \bar{X}_{N+M})^2 &= \sum_{i=1}^N (X_i^2 - 2 \bar{X}_{N+M} X_i + \bar{X}_{N+M}^2) \\[6pt] &= \sum_{i=1}^N X_i^2 - 2 \bar{X}_{N+M} \sum_{i=1}^N X_i + N \bar{X}_{N+M}^2 \\[6pt] &= \sum_{i=1}^N X_i^2 - \frac{2}{N+M} \sum_{i=1}^{N+M} \sum_{j=1}^N X_i X_j + \frac{N}{(N+M)^2} \sum_{i=1}^{N+M} \sum_{j=1}^{N+M} X_i X_j. \\[6pt] \end{align}$$

Assuming your data is IID with mean $\mu$ and variance $\sigma^2$ you have:

$$\begin{align} \mathbb{E} \bigg( \sum_{i=1}^N X_i^2 \bigg) &= N (\sigma^2 + \mu^2), \\[6pt] \mathbb{E} \bigg( \sum_{i=1}^{N+M} \sum_{j=1}^N X_i X_j \bigg) &= N [\sigma^2 + (N+M) \mu^2], \\[6pt] \mathbb{E} \bigg( \sum_{i=1}^{N+M} \sum_{j=1}^{N+M} X_i X_j \bigg) &= (N+M) [\sigma^2 + (N+M) \mu^2]. \\[6pt] \end{align}$$

We therefore have:

$$\begin{align} \mathbb{E} \bigg( \sum_{i=1}^N (X_i - \bar{X}_{N+M})^2 \bigg) &= \mathbb{E} \bigg( \sum_{i=1}^N X_i^2 - \frac{2}{N+M} \sum_{i=1}^{N+M} \sum_{j=1}^N X_i X_j \\[6pt] &\quad \quad \quad \quad \quad + \frac{N}{(N+M)^2} \sum_{i=1}^{N+M} \sum_{j=1}^{N+M} X_i X_j \bigg) \\[6pt] &= N (\sigma^2 + \mu^2) - \frac{2N}{N+M} [\sigma^2 + (N+M) \mu^2] \\[6pt] &\quad \quad \quad \quad \quad \quad \ \ + \frac{N}{N+M} [\sigma^2 + (N+M) \mu^2] \\[6pt] &= N (\sigma^2 + \mu^2) - \frac{N}{N+M} [\sigma^2 + (N+M) \mu^2] \\[6pt] &= N \sigma^2 + N \mu^2 - \frac{N \sigma^2}{N+M} - N \mu^2 \\[6pt] &= \bigg( N - \frac{N}{N+M} \bigg) \sigma^2 \\[6pt] &= N \bigg( 1 - \frac{1}{N+M} \bigg) \sigma^2 \\[6pt] &= \frac{N (N+M-1)}{N+M} \cdot \sigma^2. \\[6pt] \end{align}$$

Consequently, we obtain the following unbiased estimator for $\sigma^2$:

$$S_*^2 \equiv \frac{N+M}{N (N+M-1)} \sum_{i=1}^N (X_i - \bar{X}_{N+M})^2.$$

In the special case where $M=0$ this reduces to the usual sample variance estimator.

Answer 2

This can be worked out directly. We seek to find $P$ such that if sample variance is defined as

$$\hat{\sigma}^2 = \frac{1}{P} \sum^M_{i=1} (x_i - \frac{1}{N}\sum^{N}_{i=1} x_i)^2 $$

with $N \geq M$ then

$$E(\hat{\sigma}^2) = Var(X)$$

Answer: after some working out, what I get is that you should choose $P$ as

$$P = M \cdot \frac{N-1}{N}$$

For $M = N$, this is $N - 1$ as expected, and for $N = 100$, $M=50$, this is $49.5$ (nicely equal to the average of the cases with 0 and 1 missing degrees of freedom).