I'm studying Bayesian decision theory as introduction to machine learning and I see the concept of Risk in a lot of places. In the course I read, they define risk as:

Risk is the expected error measured by a loss function.

Which makes me think, why do we need this special construct of Risk? Isn't loss function already denoting the error?

Same with Bayes Risk, which is the lowest of all errors. Why not call it lowest loss possible?

Am I missing something here?

  • 4
    $\begingroup$ The word "expected" is the key difference here. Loss is measured at some specific dataset, while expected loss averages over different possible datasets that you might encounter. For example, if you play darts and we assume that you always go for 60 points, then your loss at one particular very good throw may be zero, but it is unlikely that, when averaging over all your possible throws, your expected loss would still be zero, as you will also (in fact, probably, mostly) miss your target by some distance. $\endgroup$ Sep 21, 2021 at 13:02
  • 2
    $\begingroup$ A non-Bayesian analysis tends to be much more complicated than a Bayesian analysis in this regard, because the former has to compare risk curves rather than just two expected risks. $\endgroup$
    – whuber
    Sep 21, 2021 at 13:09

1 Answer 1


How would you define "lowest loss possible"?

We are talking here about the possible actions $a$ given the current state of environment $\theta$, those actions may lead to some rewards $r$ where each reward is characterized with a utility $U(r)$. The rewards that are possible, can be considered as a random variable $Z$, which has a distribution that depends on $a$ and $\theta$. We usually look at the expected value of $Z$

$$ U(a, \theta) = E_{a, \theta}[U(Z)] $$

and our aim s usually to find an action $a$ that maximizes the expected utility

$$ \underset{a}{\operatorname{arg\max}} \; U(a, \theta) $$

In plain English: we are looking for such action that on average give us the best outcome. In fact, the whole idea of expected value arose from considering such scenarios. Loss is defined as a negative utility $L(a, \theta) = -U(a, \theta)$, so instead of maximizing utility, we can minimize a loss, they are the same thing. So you need to consider all the components: the context $\theta$, the actions $a$, and the rewards $Z$ associated with them. This is where we need to make it more precise what we mean by "the lowest possible".

To give an example, let's say that you are lost in the middle of the desert and need to choose between two actions: drinking water from a bottle you found or not doing it and waiting for help thirsty. None of the actions has a certain outcome. If you choose to drink, it may give you extra hours to prolong dehydration. On another hand, there is a chance that the water in the bottle is polluted or poisoned, in such a case you may dehydrate faster, or even die straightaway. If you choose not to drink, your health would gradually deteriorate, you may or may not, be able to survive long enough to get help. What action would you choose? If you are risk-averse and look only at the worst possible outcome then both cases are equally bad: you die in pain. If you are an optimist, the best possible outcome of each of the both actions is that you survive (yay!). But does any of the strategies help to make a decision? If you think about it, it is more reasonable to weigh the possible losses by their probabilities (how likely is it that the water is toxic? how likely are you to die without any water?). If you have a high chance of dying without water and a moderate chance of dying after drinking the water, maybe you should bet for the water? (It's a made-up example, not desert survival advice.) This is what expected value does.

I'm not saying that expected loss is the only thing you should ever worry about. There are cases where you would use the minimax strategy. However in many cases, expected loss is a reasonable thing to consider. It shrinks the distribution of the possible utilities into a single number that can be used for decision-making or optimization. This is especially important if you do the optimization algorithmically so that you need a single-number criterion to optimize. If you want to read more about expected utility, I can recommend the great Why care so much about expected utility? thread. For more details you can also check the Statistical Decision Theory and Bayesian Analysis book by James O. Berger.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.