# Loss functions encode our objectives for the learning system. How is this encoding guaranteed?

I understand that loss functions encode how we want our learning system to behave in a sense. For example, take a binary classification task. There are two types of error: False negatives, and false positives.

Claim 1: If our error function penalizes false negatives more than false positives, then after learning our system will be more likely to make the error of a false positive.

Is this claim true?

And if so, how does this encoding happen exactly? I feel that it's an automatic consequence of minimizing the loss function but what's going on behind the scenes?

• More likely than what? It is presumably correct that that an error function penalises false negatives more than false positives would typically have more false positives than an error function which penalised them equally. But there will be many examples of where an error function which penalises false negatives more than false positives would still have more false negatives than false positives. Jun 27 at 23:33
• I see. What would make it so that an "error function which penalises false negatives more than false positives would still have more false negatives than false positives"? Jun 28 at 9:11
• As an example, suppose you have a situation where there are many many more more actual negatives than positives and your predictor is not very skilful. Then a positive prediction is much more likely to be wrong than a negative prediction and the sensible thing is to always predict negative even if the penalty for false negatives its higher - you will make many fewer of them. Jun 28 at 9:24
• Forgive me if this is naive, but what then is the point of specifying an error function that penalizes one type of error more than another in terms of system behaviour? Jun 28 at 10:20
• Because you want the loss function to lead to optimal decisions rather than particular outcomes. If a false negative is five times as costly as a false positive then you would not want to reduce the expected number of false negatives by $1$ if this led to an expected increase in the the expected number of false positives by $100$, but you would if it led to an expected increase in the the expected number of false positives by $2$ Jun 28 at 10:35

For example, although probably not the best option, a loss function of the following form (let $$\hat y$$ be a sigmoid output) will enable some basic mechanism to choose between the two: $$L(y,\hat y) = \begin{cases}y > \hat y && \lambda (y-\hat y)^2\\ y \leq \hat y&& (y-\hat y)^2\end{cases}$$
If $$\lambda>1$$, the error will be larger when $$y=1$$, so the optimization will try to make as little error as possible when $$y=1$$, e.g. trying to decrease false negatives, this can result in making more false positives. For example, if $$\lambda\rightarrow \infty$$, the best option is to choose $$\hat y=1$$ always, making a lot of false positive errors.