# How about evaluating an estimator using the VARIANCE of loss (instead of the expectation of loss)?

The risk of an estimator $$\delta$$ is defined as $$E_\theta[L(\theta,\delta(X))],$$ where, say, $$L(\theta,\delta(X)) = (\theta-\delta(X))^2$$, and $$E_\theta(X)$$ is defined as $$\int XdP_\theta$$, namely the expectation of random variable $$X$$ when the parameter is $$\theta$$.

I wonder why people never considered $$Var_\theta[L(\theta,\delta(X))]$$ in addition to the risk as a way to evaluate an estimator given a predefined loss function?

Say $$\delta_1$$ and $$\delta_2$$ have similar risk, but different variance of loss. Intuitively I would choose the one with a smaller variance of loss. However, I've never seen people ever talked about it. Have they?

• That would make a very precise but inaccurate predictor look very good. For example, if I consistently predict tomorrow's temperature to be exactly 20 degrees too high, the variance of the difference would be 0. A precise but inaccurate predictor needs to be calibrated. Nov 4, 2020 at 17:16
• @RobbytheBelgian I understand. My question is "in addition" to the expectation of loss. Suppose two estimators give the same expectation of loss for some $\theta$, then why people never consider using their variances of loss to further pick the best of these two?
– Tan
Nov 4, 2020 at 17:19
• It would essentially mean using a different loss function. By changing the loss function you can penalize occasional high errors more. Choosing the correct loss function for a given problem is not trivial. Any crucial risk assessment needs to be taken into account. Nov 4, 2020 at 17:27
• Yes, I hear you. However, we always predefined a loss before we talk about the quality of an estimator, e.g. squared loss for the risk of an estimator. We basically assess an estimator under a specific loss.
– Tan
Nov 4, 2020 at 17:35
• It's an interesting question. You would, effectively, optimize an estimator that balances the error across samples (not one that achieves minimal error overall). Perhaps you can frame it along the likes of a loglikelihood too Nov 4, 2020 at 18:12

I saw something similar in two papers:

The second one is more on point (and easier to read to me).

They consider the following problem statement: minimize the empirical risk given by $$R(\theta):=\mathbb E[\ell(\theta,x)]$$

The following provides a lower-bound, given a sample $$X$$ from the population $$x$$ and a parameter space $$\Theta$$:

$$R(\theta)\leq\mathbb E[\ell(\theta,X)] + C_1 \sqrt\frac{\operatorname{Var}(\ell(\theta,X))}{n}+\frac{C_2}{n} \quad \forall \quad \theta \text{ in } \Theta$$

According to authors, the bound can be interpreted in regards to the bias-variance tradeoff. They then use this bound to define a robustly regularized risk.

• It looks that the paper is not assessing an estimator but rather to estimate $\theta$ given $X$ using risk.
– Tan
Nov 4, 2020 at 19:28
• @Tan true, but the lower bound is still true (given the constraints), even though it's not the main point of the paper. Nov 4, 2020 at 19:30