Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm studying Bayesian decision theory as introduction to machine learning and I see the concept of Risk in a lot of places. In the course I read, they define risk as:
Risk is the expected error measured by a loss function.
Which makes me think, why do we need this special construct of Risk? Isn't loss function already denoting the error?
Same with Bayes Risk, which is the lowest of all errors. Why not call it lowest loss possible?
Am I missing something here?
How would you define "lowest loss possible"?
We are talking here about the possible actions $a$ given the current state of environment $\theta$, those actions may lead to some rewards $r$ where each reward is characterized with a utility $U(r)$. The rewards that are possible, can be considered as a random variable $Z$, which has a distribution that depends on $a$ and $\theta$. We usually look at the expected value of $Z$
$$ U(a, \theta) = E_{a, \theta}[U(Z)] $$
and our aim s usually to find an action $a$ that maximizes the expected utility
$$ \underset{a}{\operatorname{arg\max}} \; U(a, \theta) $$
In plain English: we are looking for such action that on average give us the best outcome. In fact, the whole idea of expected value arose from considering such scenarios. Loss is defined as a negative utility $L(a, \theta) = -U(a, \theta)$, so instead of maximizing utility, we can minimize a loss, they are the same thing. So you need to consider all the components: the context $\theta$, the actions $a$, and the rewards $Z$ associated with them. This is where we need to make it more precise what we mean by "the lowest possible".
To give an example, let's say that you are lost in the middle of the desert and need to choose between two actions: drinking water from a bottle you found or not doing it and waiting for help thirsty. None of the actions has a certain outcome. If you choose to drink, it may give you extra hours to prolong dehydration. On another hand, there is a chance that the water in the bottle is polluted or poisoned, in such a case you may dehydrate faster, or even die straightaway. If you choose not to drink, your health would gradually deteriorate, you may or may not, be able to survive long enough to get help. What action would you choose? If you are risk-averse and look only at the worst possible outcome then both cases are equally bad: you die in pain. If you are an optimist, the best possible outcome of each of the both actions is that you survive (yay!). But does any of the strategies help to make a decision? If you think about it, it is more reasonable to weigh the possible losses by their probabilities (how likely is it that the water is toxic? how likely are you to die without any water?). If you have a high chance of dying without water and a moderate chance of dying after drinking the water, maybe you should bet for the water? (It's a made-up example, not desert survival advice.) This is what expected value does.
I'm not saying that expected loss is the only thing you should ever worry about. There are cases where you would use the minimax strategy. However in many cases, expected loss is a reasonable thing to consider. It shrinks the distribution of the possible utilities into a single number that can be used for decision-making or optimization. This is especially important if you do the optimization algorithmically so that you need a single-number criterion to optimize. If you want to read more about expected utility, I can recommend the great Why care so much about expected utility? thread. For more details you can also check the Statistical Decision Theory and Bayesian Analysis book by James O. Berger.
