2
$\begingroup$

In this pytorch example, the output layer does not have an activation function even though the neural network is being used for a binary classification task (i.e. ground truth values are either 0 = negative or 1 = positive). After inspecting the output, I can see that there are values such as -13.02 or 4.56, which are obviously not bounded between 0 and 1. Also, after adding a sigmoid activation function in myself, the performance seems to be worse than without the activation function. Thus, I have two questions:

  1. Would 0 be the threshold that determines if the predicted output is the negative or positive class? In other words, is any output value $\hat{y}$, $ \hat{y}< 0$ is negative and $\hat{y}>= 0 $is positive?
  2. Why does not using an activation function lead to better performance when this is a binary classification problem, not a regression problem? Is it specific to this example?
$\endgroup$

1 Answer 1

1
$\begingroup$
  1. Choosing a threshold depends on what trade-off you wish to achieve in terms of classification errors. Choosing 0 or a different number could be appropriate, depending on context.
  2. The network is trained using BCEWithLogitsLoss, which combines the sigmoid activation and the binary cross-entropy loss into a single call. This is described in the pytorch documentation. Combining the two operations avoids round-tripping $\exp$ and $\log$, which can cause severe loss of precision in floating-point arithmetic.
$\endgroup$
1
  • $\begingroup$ I must have overlooked the BCEWithLogitsLoss, but now it makes sense. Cheers! $\endgroup$
    – 3michelin
    Commented Dec 29, 2020 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.