Why, *intuitively*, in regular parametric problems, does uncertainty go down at a $\sqrt{ n }$ rate on the SE/posterior SD scale?

Question

consider the simplest regular statistical inference problem: $( y_1, \dots, y_n | F ) \sim$ $\text{IID}$ from a cumulative distribution function $F$ on $\mathbb{ R }$ with mean $\mu$ and finite variance $\sigma^2$ (the finiteness of $\sigma$ ensures the existence and finiteness of $\mu$); let $\mathbf{ y } = ( y_1, \dots, y_n )$

the nonparametric maximum likelihood estimator for $\mu$, which cannot be improved upon either from the frequentist or bayesian point of view without additional information/assumptions about $F$, is of course $\bar{ y } = \frac{ 1 }{ n } \sum_{ i = 1 }^n y_i$

the repeated-sampling (RS, frequentist) variance of $\bar{ y }$ is of course $V_{ RS } ( \bar{ y } ) = \frac{ \sigma^2 }{ n }$, leading to the familiar standard error formula $SE_{ RS } ( \bar{ y } ) = \sqrt{ V_{ RS } ( \bar{ y } ) } = \frac{ \sigma }{ \sqrt{ n } }$ (with this same information base, a bayesian would of course get the same answer for the standard deviation of her/his posterior distribution for $\mu$ given $\mathbf{ y }$)

this formula, which we could call the square root law, depends vitally on the fact that the repeated-sampling variance of a sum of IID observables is the sum of their variances: the variance and sum operators commute under independence

so the issue comes down to this: why, intuitively, do we live in a universe in which it is the variance scale on which uncertainty about the sum of independent observables is additive, and not some other scale?

if i were trying to intuitively explain this fundamental fact to an intelligent person who has had little exposure to quantitative thinking, i do not regard it as satisfying (again intuitively) to offer the following argument:

(1) define a kolmogorov-style probability triple $( \Omega, \mathcal{ F }, P )$

(2) define real-valued random variables $Y_i$ as functions from sets in $\mathcal{ F }$ to $\mathbb{ R }$

(3) define the concept of expectation $E ( Y )$ of a random variable $Y$

(4) define the concept of variance $V ( Y ) = E [ Y - E ( Y ) ]^2$

(5) define the concept of independence of a finite collection $\mathbf{ Y } = ( Y_1, \dots, Y_n )$ of random variables

(6) prove that under independence, the variance of a sum of random variables equals the sum of their variances

(7) deduce the square root law from (1-6)

so, i repeat my question:

why, intuitively, do we live in a universe in which it is the variance scale on which uncertainty about the sum of independent observables is additive, and not some other scale?

cogent thoughts on this topic would interest me; thanks in advance for your interest, and best wishes

Answer 1

The intuition for the variance is best understood in geometric terms, by likening the variance to a geometric analogy. With this analogy, the intuition of the additive nature of the variance operator goes back to Pythagoras Theorem.

The natural scale for measuring variability is the standard deviation, which is a scale-preserving measure of variability. When summing independent random variables, this measure operates effectively like a vector norm in Euclidean space, with the individual dimensions being the standard deviations of the individual random variables. The variance operator operates like a squared-norm, and so it has the additive property of the Pythagoras theorem. Hence, intuitively, the square-root law in an IID model arises from the geometric properties of vectors composed of orthogonal parts.

To see this more clearly, note that moment relations for independent random variables can be represented using orthogonal vectors in Euclidean space. If $Y_1,...,Y_n$ are orthogonal vectors in Euclidean space then from Pythagoras theorem, the squared-norm obeys the following property:

$$||\dot{Y}||^2 = \sum_{i=1}^n ||Y_i||^2.$$

(And of course, it is also notable that this can be extended even to correlated random variables, in which case the squared-norm of their sum becomes a more general quadratic form of the vectors.) Hence, taking the "sample mean" (in this case an average of orthogonal vectors) for vectors with a common length $||Y|| = ||Y_i||$, gives:

$$||\bar{Y}||^2 = \frac{1}{n^2} \sum_{i=1}^n ||Y_i||^2 = \frac{1}{n} ||Y||^2.$$

If each vector has the same norm (which is analogous to the common variance in the IID model) then the square-root law $||\bar{Y}|| = ||Y||/\sqrt{n}$ holds geometrically from the additive nature of the squared-norm.