consider the simplest regular statistical inference problem: $( y_1, \dots, y_n | F ) \sim$ $\text{IID}$ from a cumulative distribution function $F$ on $\mathbb{ R }$ with mean $\mu$ and finite variance $\sigma^2$ (the finiteness of $\sigma$ ensures the existence and finiteness of $\mu$); let $\mathbf{ y } = ( y_1, \dots, y_n )$

the nonparametric maximum likelihood estimator for $\mu$, which cannot be improved upon either from the frequentist or bayesian point of view without additional information/assumptions about $F$, is of course $\bar{ y } = \frac{ 1 }{ n } \sum_{ i = 1 }^n y_i$

the repeated-sampling (RS, frequentist) variance of $\bar{ y }$ is of course $V_{ RS } ( \bar{ y } ) = \frac{ \sigma^2 }{ n }$, leading to the familiar standard error formula $SE_{ RS } ( \bar{ y } ) = \sqrt{ V_{ RS } ( \bar{ y } ) } = \frac{ \sigma }{ \sqrt{ n } }$ (with this same information base, a bayesian would of course get the same answer for the standard deviation of her/his posterior distribution for $\mu$ given $\mathbf{ y }$)

this formula, which we could call the square root law, depends vitally on the fact that the repeated-sampling variance of a sum of IID observables is the sum of their variances: the variance and sum operators commute under independence

so the issue comes down to this: why, intuitively, do we live in a universe in which it is the variance scale on which uncertainty about the sum of independent observables is additive, and not some other scale?

if i were trying to intuitively explain this fundamental fact to an intelligent person who has had little exposure to quantitative thinking, i do not regard it as satisfying (again intuitively) to offer the following argument:

(1) define a kolmogorov-style probability triple $( \Omega, \mathcal{ F }, P )$

(2) define real-valued random variables $Y_i$ as functions from sets in $\mathcal{ F }$ to $\mathbb{ R }$

(3) define the concept of expectation $E ( Y )$ of a random variable $Y$

(4) define the concept of variance $V ( Y ) = E [ Y - E ( Y ) ]^2$

(5) define the concept of independence of a finite collection $\mathbf{ Y } = ( Y_1, \dots, Y_n )$ of random variables

(6) prove that under independence, the variance of a sum of random variables equals the sum of their variances

(7) deduce the square root law from (1-6)

so, i repeat my question:

why, intuitively, do we live in a universe in which it is the variance scale on which uncertainty about the sum of independent observables is additive, and not some other scale?

cogent thoughts on this topic would interest me; thanks in advance for your interest, and best wishes

  • Many thanks David. However, now it's easy to read the question, I'm not quite clear on the kind of answer you seek (only on one kind of answer you don't seek) – Glen_b Sep 22 at 0:58
  • I have made a major edit to make the question shorter and simple, and simplify the title. Please check that this edit correctly specifies your question. Feel free to undo if this change alters your question. – Ben Sep 22 at 1:06
  • ben, i would indeed like to undo your major edit; how do i do that? – David Draper Sep 22 at 3:48
  • 1
    @David Draper: I have rolled back the edit at your request. – Ben Sep 22 at 5:10

The intuition for the variance is best understood in geometric terms, by likening the variance to a geometric analogy. With this analogy, the intuition of the additive nature of the variance operator goes back to Pythagoras Theorem.

The natural scale for measuring variability is the standard deviation, which is a scale-preserving measure of variability. When summing independent random variables, this measure operates effectively like a vector norm in Euclidean space, with the individual dimensions being the the standard deviations of the individual random variables. The variance operator operates like a squared-norm, and so it has the additive property of the Pythagoras theorem. Hence, intuitively, the square-root law in an IID model arises from the geometric properties of vectors composed of orthogonal parts.

To see this more clearly, note that moment relations for independent random variables can be represented using orthogonal vectors in Euclidean space. If $Y_1,...,Y_n$ are orthogonal vectors in Euclidean space then from Pythagoras theorem, the squared-norm obeys the following property:

$$||\dot{Y}||^2 = \sum_{i=1}^n ||Y_i||^2.$$

(And of course, it is also notable that this can be extended even to correlated random variables, in which case the squared-norm of their sum becomes a more general quadratic form of the vectors.) Hence, taking the "sample mean" (in this case an average of orthogonal vectors) for vectors with a common length $||Y|| = ||Y_i||$, gives:

$$||\bar{Y}||^2 = \frac{1}{n^2} \sum_{i=1}^n ||Y_i||^2 = \frac{1}{n} ||Y||.$$

If each vector has the same norm (which is analogous to the common variance in the IID model) then the square-root law $||\bar{Y}|| = ||Y||/\sqrt{n}$ holds geometrically from the additive nature of the squared-norm.

  • ben, thanks for your interesting comment -- of course you've just kicked the can down the road, by transferring the intuitive burden from the world of random variables to the world of normed inner product spaces -- if you consider the amount of detail that you would have to set up with your approach to get to the desired result, PLUS the whole $( \Omega, \mathcal{ F }, P )$ story to get random variables up and running, AND then building the bridge between the inner product space and random variables, you have hardly decreased the intuitive burden ... – David Draper Sep 22 at 4:04
  • 2
    If that is the standard, then it is unclear how you could ever reduce the intuitive burden. That advantage of working with vector norms is that it yields geometric insights that you can directly see (e.g., by looking at graphical depictions of Pythagoras theorem). If that does nothing to help with the intuition then I can't see what could possible assist you. – Ben Sep 22 at 5:08
  • @David I believe this relates to the difficulty I alluded to in my comment under the question. Superficially it seems to admit all manner of answers (indeed, calls for 'cogent thoughts' is a request for discussion, making it overly broad by SE's standards), but if answers are rejectable on incompletely specified criteria (while narrowing the scope somewhat), it instead becomes unclear (since we're partly left to try to guess what would be acceptable). Either way, it seems like it might be more of a discussion item than a question with a correct answer, ... ctd – Glen_b Sep 22 at 7:59
  • ctd... and so perhaps more suitable on the chat in its present form. If you're able to edit it to make it a question that has a clear scope with clear criteria for an acceptable answer it might remain, but otherwise I expect it's likely to be put on hold as either too broad or unclear. – Glen_b Sep 22 at 8:02
  • 1
    many thanks to ben and glen -- it's ok with me to put this on hold – David Draper Sep 22 at 15:00

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.