1
$\begingroup$

I understand that loss functions encode how we want our learning system to behave in a sense. For example, take a binary classification task. There are two types of error: False negatives, and false positives.

Claim 1: If our error function penalizes false negatives more than false positives, then after learning our system will be more likely to make the error of a false positive.

Is this claim true?

And if so, how does this encoding happen exactly? I feel that it's an automatic consequence of minimizing the loss function but what's going on behind the scenes?

$\endgroup$
6
  • 1
    $\begingroup$ More likely than what? It is presumably correct that that an error function penalises false negatives more than false positives would typically have more false positives than an error function which penalised them equally. But there will be many examples of where an error function which penalises false negatives more than false positives would still have more false negatives than false positives. $\endgroup$
    – Henry
    Jun 27 at 23:33
  • $\begingroup$ I see. What would make it so that an "error function which penalises false negatives more than false positives would still have more false negatives than false positives"? $\endgroup$ Jun 28 at 9:11
  • 1
    $\begingroup$ As an example, suppose you have a situation where there are many many more more actual negatives than positives and your predictor is not very skilful. Then a positive prediction is much more likely to be wrong than a negative prediction and the sensible thing is to always predict negative even if the penalty for false negatives its higher - you will make many fewer of them. $\endgroup$
    – Henry
    Jun 28 at 9:24
  • $\begingroup$ Forgive me if this is naive, but what then is the point of specifying an error function that penalizes one type of error more than another in terms of system behaviour? $\endgroup$ Jun 28 at 10:20
  • 1
    $\begingroup$ Because you want the loss function to lead to optimal decisions rather than particular outcomes. If a false negative is five times as costly as a false positive then you would not want to reduce the expected number of false negatives by $1$ if this led to an expected increase in the the expected number of false positives by $100$, but you would if it led to an expected increase in the the expected number of false positives by $2$ $\endgroup$
    – Henry
    Jun 28 at 10:35
1
$\begingroup$

Yes, your claim is correct given that system converged to a good local minimum. As you mentioned, this is an automatic consequence of the loss function. The encoding is in the loss function. Behind the scenes, it's just the minimization procedure, e.g. gradient descent, that makes it real.

For example, although probably not the best option, a loss function of the following form (let $\hat y$ be a sigmoid output) will enable some basic mechanism to choose between the two: $$L(y,\hat y) = \begin{cases}y > \hat y && \lambda (y-\hat y)^2\\ y \leq \hat y&& (y-\hat y)^2\end{cases}$$

If $\lambda>1$, the error will be larger when $y=1$, so the optimization will try to make as little error as possible when $y=1$, e.g. trying to decrease false negatives, this can result in making more false positives. For example, if $\lambda\rightarrow \infty$, the best option is to choose $\hat y=1$ always, making a lot of false positive errors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.