Consider the risk function R of an estimator (statistic) $\delta(X)$ trying to estimate parameter $\theta$:

$$R(\theta, \delta) = E_{X \sim P_{\theta}}[Loss(\theta,\delta(X)]$$

Which can be interpreted as the average loss our estimator has weighted by relative frequency of the data X.

Then consider Bayes risk:

$$R(\delta, \lambda) = \int_{\Omega}R(\theta, \delta) \lambda(\theta) d\theta$$

where $\lambda(\theta)$ is a weighting function on each possible value the parameter could take.

How would one interpret $R(\delta, \lambda)$?

The thing that is kind of confusing me is that because we are integrating with respect to both X and $\theta$, it was harder for me to interpret what it means.

The interpretation I was thinking was something like this:

The "real" (average) loss of $\delta$. Why real? Well, because we are using $\lambda(\theta)$ to put a higher weights on which parameters we care more about. We want to have an estimator that is good in general but we care more about some $\theta$'s, depending on $\lambda$.

  • 2
    $\begingroup$ If $\lambda(\theta)$ is such that $\int \lambda d\theta = 1$ and $\lambda > 0$, then you can regard $\lambda(\theta)$ as a prior. then $R(\delta, \lambda)$ is the marginal expected risk under of $\delta$ the prior $\lambda$. Hence the name "Bayes risk", because it is the marginal expected loss from the Bayesian point of view. Granted, this may be unsatisfying since the question is tagged "frequentist", but this is certainly the intuition behind the quantity (and a frequentist can care about it despite this interpretation). $\endgroup$
    – guy
    Feb 8 '15 at 2:18
  • 1
    $\begingroup$ @guy what you said is not correct. Look at the end of the following: s3.amazonaws.com/piazza-resources/i5e65wn4vyf32u/i5qwx6cynyt639/… hence the frequentist tag on my question and the explicit absence of bayesianism in the discussion of my question. (last page btw). Apparently, I was told that treating the data as a r.v. is not at all a bayesian thing to do. Not sure why though. $\endgroup$ Feb 8 '15 at 2:31
  • 2
    $\begingroup$ It is a Bayesian notion before observing any data. So, I say, given this prior and before collecting any data, what is my expected loss? For example, I am a manager confronted with some decision problem, and I want to know before ordering an experiment to be conducted a measure of average loss. From a Bayesian viewpoint, obviously once I collect the data I have no reason to average over $X$; rather, I should condition on it.This is why I emphasized that it is the marginal risk, i.e. averaging over the data; a Bayesian might do this before he has actually observed the data. $\endgroup$
    – guy
    Feb 10 '15 at 15:44

The Bayes risk is the frequentist risk averaged over the parameter space against to the prior distribution $\lambda$. The notion turns a function of $\theta$, $R(\theta,\delta)$, into a positive number, $R(\delta,\lambda)$, and hence allows for a total ordering of estimators $\delta$, hence for the definition of the Bayes estimator$$\delta^\lambda=\arg\min_\delta R(\delta,\lambda)$$

The link with the conditional Bayesian approach is that, thanks to Fubini's theorem, the Bayes estimator can also be derived by minimising for every $x$ the posterior expected loss$$\varrho(\delta(x),\lambda)=\mathbb{E}[L(\theta,\delta(x))|X=x]$$

As noted by @guy, this quantity also has frequentist motivations, one of them being that, in regular problems, the minimax risk equals the maximin risk in the following way:$$\min_\delta\max_\theta R(\theta,\delta)=\max_\lambda\min_\delta R(\delta,\lambda)$$expressing minimax estimators as "worst" Bayes estimators.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.