minimum number of points to calculate variance

Question

It seems to be that the minimum number of observations needed to calculate variance is 2. I can see the logic behind this because by logic, there can not be variance for a single point. But on the other hands, I don't think 2 points are really enough to calculate variance in general. This is because you need to observe more data to understand how much variance there truly is.

Ex: if it rains 2 days but does not rain 1 day, the variance might tell me that there is a a lot of variability in rain, but more data would really be needed to be certain. This is actually why you need something like min 25-30 data points to calculate the variance.

So if this is true, how can we have variance in repeated measure longitudinal regression models when there is usually just 3-4 obs max per person? If you combine everything into one dataset, I think you might be able to calculate variance for whole population .... but how can you calculate individual variance for each person individually when you only have 3-4 obs? Isn't it like the rain example?

Answer 1

One can argue that you only need $n=1$ to estimate variance, except that the variance is always estimated to be $0$ when you calculate $E[X^2] - E[X]^2 = X_1^2 - X_1^2 = 0$ and that this is a biased estimator.

"Mixed models" comprise a large, large class of models. In the simplest example: using random intercepts, it's relatively straightforward to understand why the estimates are reliable in moderate sample sizes (even with "3-4 obs max per person"). The mixed model uses a complicated estimation approach called profile likelihood which has the desirable property of shrinkage. For instance, the Neyman-Scott problem illustrates why maximum likelihood breaks down when the number of parameters is large. Stated briefly, Neyman-Scott concerns the problem of estimating the variance using maximum likelihood in a process of IID dyads having $$ [X_{i,1}, X_{i,2}] \sim \mathcal{N}(\mu_i, \sigma^2 \mathcal{I})$$

More background here but the asymptotic MLE for $\sigma^2 $ $\rightarrow$ 2$\sigma^2$ as $n \rightarrow \infty$. The mixed model approach would model the $\mu_i$ as a normal distribution and use the quantiles as plugin estimates for the $\mu_i$ before maximizing the joint normal likelihood as an estimator for $\sigma^2$. Ridge estimators also enforce shrinkage. The specific estimates of $\mu_i$ are biased a little but the estimates for $\sigma^2$ have far more favorable small and large sample properties. It is precisely because of this distributional modeling that the number of observations per cluster can be extremely small and yet the estimators of fixed effects and variance structures have highly favorable properties.

In spite of this, there are many missteps in the literature and elsewhere where the wrong mixed model is applied and the variance components are subject to sparse estimation. Here, just as in any ordinary OLS, unreliable estimates lead to unreliable inference, and much of it can be missed without doing thorough diagnostics.