What problem or game are variance and standard deviation optimal solutions for?

Question

For a given random variable (or a population, or a stochastic process), mathematical expectation is the answer to a question What point forecast minimizes the expected square loss?. Also, it is the optimal solution to a game Guess the next realization of a random variable (or a new draw from a population), and I will punish you by the squared distance between the value and your guess if you have linear disutility in terms of the punishment. Median is the answer to a corresponding question under absolute loss and mode is the answer under "all or nothing" loss.

Questions: Does variance and standard deviation answer any similar questions? What are they?

The motivation for this question stems from teaching basic measures of central tendency and spread. Whereas the measures of central tendency can be motivated by decision-theoretic problems above, I wonder how one could motivate the measures of spread.

Answer 1

If I have understood the question as intended, you have in mind a setting in which you can obtain independent realizations of any random variable $X$ with any distribution $F$ (having finite variance $\sigma^2(F)$). The "game" is determined by functions $h$ and $\mathcal L$ to be described. It consists of the following steps and rules:

1. Your opponent ("Nature") reveals $F.$

2. In response you produce a number $t(F),$ your "prediction."

To evaluate the outcome of the game, the following calculations are performed:

• A sample of $n$ iid observations $\mathbf{X}=X_1, X_2, \ldots, X_n$ is drawn from $F.$

• A predetermined function $h$ is applied to the sample, producing a number $h(\mathbf{X}),$ the "statistic."

• The "loss function" $\mathcal{L}$ compares your "prediction" $t(F)$ to the statistic $h(\mathbf{X}),$ producing a non-negative number $\mathcal{L}(t(F), h(\mathbf{X})).$

• The outcome of the game is the expected loss (or "risk") $$R_{(\mathcal{L}, h)}(t, F) = E(\mathcal{L}(t(F), h(\mathbf{X}))).$$

Your objective is to respond to Nature's move by specifying some $t$ that minimizes the risk.

For example, in the game with the function $h(X_1)=X_1$ and any loss of the form $\mathcal{L}(t, h) = \lambda(t-h)^2$ for some positive number $\lambda,$ your optimal move is to pick $t(F)$ to be the expectation of $F.$

The question before us is,

Do there exist $\mathcal{L}$ and $h$ for which the optimal move is to pick $t(F)$ to be the variance $\sigma^2(F)$?

This is readily answered by exhibiting the variance as an expectation. One way is to stipulate that $$h(X_1,X_2) = \frac{1}{2}(X_1-X_2)^2$$ and continue to use quadratic loss $$\mathcal{L}(t,h) = (t-h)^2.$$ Upon observing that

$$E(h(\mathbf{X})) = \sigma^2(F),$$

the example allows us to conclude that this $h$ and this $\mathcal L$ answer the question about variance.

---

What about the standard deviation $\sigma(F)$? Again, we need only exhibit this as the expectation of a sample statistic. However, that's not possible, because even when we limit $F$ to the family of Bernoulli$(p)$ distributions we can only obtain unbiased estimators of polynomial functions of $p,$ but $\sigma(F) = \sqrt{p(1-p)}$ is not a polynomial function on the domain $p\in (0,1).$ (See For the binomial distribution, why does no unbiased estimator exist for $1/p$? for the general argument about Binomial distributions, to which this question can be reduced after averaging $h$ over all permutations of the $X_i.$)