How about evaluating an estimator using the VARIANCE of loss (instead of the expectation of loss)?

Question

The risk of an estimator $\delta$ is defined as $$E_\theta[L(\theta,\delta(X))],$$ where, say, $L(\theta,\delta(X)) = (\theta-\delta(X))^2$, and $E_\theta(X)$ is defined as $\int XdP_\theta$, namely the expectation of random variable $X$ when the parameter is $\theta$.

I wonder why people never considered $$Var_\theta[L(\theta,\delta(X))]$$ in addition to the risk as a way to evaluate an estimator given a predefined loss function?

Say $\delta_1$ and $\delta_2$ have similar risk, but different variance of loss. Intuitively I would choose the one with a smaller variance of loss. However, I've never seen people ever talked about it. Have they?

Answer 1

I saw something similar in two papers:

• A Maurer & M Pontil. Empirical Bernstein Bounds and Sample Variance Penalization. 2009
• JC Duchi & H Namkoong. Variance-based regularization with convex objectives. 2017

The second one is more on point (and easier to read to me).

They consider the following problem statement: minimize the empirical risk given by $$R(\theta):=\mathbb E[\ell(\theta,x)]$$

The following provides a lower-bound, given a sample $X$ from the population $x$ and a parameter space $\Theta$:

$$R(\theta)\leq\mathbb E[\ell(\theta,X)] + C_1 \sqrt\frac{\operatorname{Var}(\ell(\theta,X))}{n}+\frac{C_2}{n} \quad \forall \quad \theta \text{ in } \Theta$$

According to authors, the bound can be interpreted in regards to the bias-variance tradeoff. They then use this bound to define a robustly regularized risk.