Minimizing square loss results in predicting conditional means.
Minimizing absolute loss results in predicting conditional medians.
What loss function results in predicting conditional variances?
I have seen this answer, which works if we can assume a conditional mean of zero (which might be a safe assumption for much of my particular application), but what if we can't assume a conditional mean of zero?
In econometrics, people have been using GARCH to model the conditional variances of time series with various degrees of success. The basic idea there is to impose a model for the conditional variances --- i.e. trying to "predict" them with known information. The parameters involved in the model for the conditional variance can then be estimated through various techniques using the log-likelihood as the objective.
I guess one can probably extend this line of thought and apply it to non-time-series data.
In short: In the general situation where the mean is not restricted to be a known constant, there exists no strictly consistent loss function for the variance. Let's unpack what that means.
The definition of (strict) consistency provides a suitable mathematical tool to interpret the statement 'Minimizing square loss results in predicting conditional means.' Essentially, a loss function $L: \mathbb{R} \times \mathbb{R} \to \mathbb{R}$ is consistent for the mean relative to the the class of distributions $\mathcal{F}$ if $$ \mathbb{E}_{Y \sim F} L( \mathbb{E}_{Y \sim F} (Y), Y) \le \mathbb{E}_{Y \sim F} L( x, Y) $$ for all $x \in \mathbb{R}$ and $F \in \mathcal{F}$. It is strictly consistent if equality implies $x= \mathbb{E}_{Y \sim F} (Y)$. The true expectation is thus the (unique) minimizer of the expected loss. Standard arguments show that squared error $L(x,y) = (x -y)^2$ satisfies this definition.
Naturally, we can investigate consistency for any statistic which can be computed from a distribution $F$, e.g. for the median, by simply replacing $\mathbb{E}_F (Y)$ in this definition by our statistic of interest. Also, nothing changes when we use conditional expectations instead of unconditional ones, so we can simply omit the conditioning for better understanding.
The question is now whether there is a loss function $L$ such that the true variance minimizes it in expectation. Luckily, theory on strictly consistent loss functions shows that they can only exist, if the statistic of interest, let's call it $T$, satisfies the convex level sets property. This states that if two distributions $F_0$ and $F_1$ satisfy $T(F_0) = T(F_1)$, then for their convex combination $F_\lambda := \lambda F_1 + (1- \lambda) F_0$ we must have $T(F_\lambda) = T(F_0)$ for all $\lambda \in [0,1]$.
The mean ($T(F) := \mathbb{E}_{Y\sim F} Y$) and also the median satisfy this condition. The variance also satisfies this condition as long as the mean is the same for $F_0$ and $F_1$. However, in convex classes $\mathcal{F}$ where the mean is not constant, the variance will violate this property for infinitely many distribution pairs. Consequently, no strictly consistent loss function for the variance can exist relative to such general classes.
Strictly consistent loss functions are often called strictly consistent scoring functions in the literature, since the concept is closely connected to (strict) propriety of scoring rules, see also my answer to Equivalent of proper scoring rule for point forecasts. The statistics which possess strictly consistent scoring/loss functions are usually called elicitable.
Convex level sets are only necessary and not sufficient for existence, i.e. a statistic which has convex level sets does not need to have a strictly consistent scoring/loss function.
Gneiting's Making and evaluating point forecasts defines (strict) consistency and gives a proof why convex level sets are necessary for their existence (Theorem 6)
