# Neural networks - Switching loss function during training for better gradients

I'm training a neural network $D$ for binary classification using binary cross entropy loss (where $y_i$ is either 1 or 0, and $D(x)$ produces a value in $[0,1]$):

$$-y_i\log(D(x_i)) - (1 - y_i)\log(1 - D(x_i))$$

Early during training, when the classification performance is relative bad, this loss function gives good gradients. However, as performance increases, the gradients get progressively worse. For a discriminator D which performs well, the following loss function produces better gradients:

$$y_i\log(1-D(x_i)) + (1 - y_i)\log(D(x_i))$$

Note that both loss functions have their minimum at the same value of $D(x)$.

My question is:

• Once $D$ reaches a certain performance, can I switch out the loss functions to get better gradients?
• Has this been done before?
• Would it be worth it?
• I'll just address the first part of the first question "can you?". Yes you can. In this case, whatever you are doing with partially solving the problem with the first loss function, is being used a method to generate a starting point for solving a different optimization problem, i.e., with the 2nd loss function. – Mark L. Stone Sep 3 '16 at 12:46