5
$\begingroup$

The risk of an estimator $\delta$ is defined as $$E_\theta[L(\theta,\delta(X))],$$ where, say, $L(\theta,\delta(X)) = (\theta-\delta(X))^2$, and $E_\theta(X)$ is defined as $\int XdP_\theta$, namely the expectation of random variable $X$ when the parameter is $\theta$.

I wonder why people never considered $$Var_\theta[L(\theta,\delta(X))]$$ in addition to the risk as a way to evaluate an estimator given a predefined loss function?

Say $\delta_1$ and $\delta_2$ have similar risk, but different variance of loss. Intuitively I would choose the one with a smaller variance of loss. However, I've never seen people ever talked about it. Have they?

Improve this question
$\endgroup$
5
  • $\begingroup$ That would make a very precise but inaccurate predictor look very good. For example, if I consistently predict tomorrow's temperature to be exactly 20 degrees too high, the variance of the difference would be 0. A precise but inaccurate predictor needs to be calibrated. $\endgroup$ – Robby the Belgian Nov 4 '20 at 17:16
  • $\begingroup$ @RobbytheBelgian I understand. My question is "in addition" to the expectation of loss. Suppose two estimators give the same expectation of loss for some $\theta$, then why people never consider using their variances of loss to further pick the best of these two? $\endgroup$ – Tan Nov 4 '20 at 17:19
  • $\begingroup$ It would essentially mean using a different loss function. By changing the loss function you can penalize occasional high errors more. Choosing the correct loss function for a given problem is not trivial. Any crucial risk assessment needs to be taken into account. $\endgroup$ – Robby the Belgian Nov 4 '20 at 17:27
  • $\begingroup$ Yes, I hear you. However, we always predefined a loss before we talk about the quality of an estimator, e.g. squared loss for the risk of an estimator. We basically assess an estimator under a specific loss. $\endgroup$ – Tan Nov 4 '20 at 17:35
  • $\begingroup$ It's an interesting question. You would, effectively, optimize an estimator that balances the error across samples (not one that achieves minimal error overall). Perhaps you can frame it along the likes of a loglikelihood too $\endgroup$ – Firebug Nov 4 '20 at 18:12
1
$\begingroup$

I saw something similar in two papers:

The second one is more on point (and easier to read to me).

They consider the following problem statement: minimize the empirical risk given by $$R(\theta):=\mathbb E[\ell(\theta,x)]$$

The following provides a lower-bound, given a sample $X$ from the population $x$ and a parameter space $\Theta$:

$$R(\theta)\leq\mathbb E[\ell(\theta,X)] + C_1 \sqrt\frac{\operatorname{Var}(\ell(\theta,X))}{n}+\frac{C_2}{n} \quad \forall \quad \theta \text{ in } \Theta$$

According to authors, the bound can be interpreted in regards to the bias-variance tradeoff. They then use this bound to define a robustly regularized risk.

Improve this answer
$\endgroup$
2
  • $\begingroup$ It looks that the paper is not assessing an estimator but rather to estimate $\theta$ given $X$ using risk. $\endgroup$ – Tan Nov 4 '20 at 19:28
  • $\begingroup$ @Tan true, but the lower bound is still true (given the constraints), even though it's not the main point of the paper. $\endgroup$ – Firebug Nov 4 '20 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.