I'm training a neural network $D$ for binary classification using binary cross entropy loss (where $y_i$ is either 1 or 0, and $D(x)$ produces a value in $[0,1]$):
$$-y_i\log(D(x_i)) - (1 - y_i)\log(1 - D(x_i))$$
Early during training, when the classification performance is relative bad, this loss function gives good gradients. However, as performance increases, the gradients get progressively worse. For a discriminator D which performs well, the following loss function produces better gradients:
$$y_i\log(1-D(x_i)) + (1 - y_i)\log(D(x_i))$$
Note that both loss functions have their minimum at the same value of $D(x)$.
My question is:
- Once $D$ reaches a certain performance, can I switch out the loss functions to get better gradients?
- Has this been done before?
- Would it be worth it?