I'm training a neural network $D$ for binary classification using binary cross entropy loss (where $y_i$ is either 1 or 0, and $D(x)$ produces a value in $[0,1]$):

$$-y_i\log(D(x_i)) - (1 - y_i)\log(1 - D(x_i))$$

Early during training, when the classification performance is relative bad, this loss function gives good gradients. However, as performance increases, the gradients get progressively worse. For a discriminator D which performs well, the following loss function produces better gradients:

$$y_i\log(1-D(x_i)) + (1 - y_i)\log(D(x_i))$$

Note that both loss functions have their minimum at the same value of $D(x)$.

My question is:

  • Once $D$ reaches a certain performance, can I switch out the loss functions to get better gradients?
  • Has this been done before?
  • Would it be worth it?
  • 1
    $\begingroup$ I'll just address the first part of the first question "can you?". Yes you can. In this case, whatever you are doing with partially solving the problem with the first loss function, is being used a method to generate a starting point for solving a different optimization problem, i.e., with the 2nd loss function. $\endgroup$ Sep 3, 2016 at 12:46

1 Answer 1

  1. yes you can do it. It's your network, anything you can code, you can do to it. As was mentioned in the comments, it's just getting a starting point for different optimization problem. Your errors having the same minimum, doesn't matter that much, because the whole problem is nonconvex and you might get stuck in different local minimals/plateaus.

  2. did you hear about pre-training? Although I don't know about any published network that would use this method to get faster convergence, it's common to pre-train the network in some way before you actually train it with your final loss function. But usually it is done in unsupervised way.

  3. You just answered yourself. It's getting you a better gradients, so it's worth it with respect to gradients. The question is, if you can get better results by doing something else. E.g: using momentum, changing your alpha on the way or as was mentioned before, to use some pre-training methods


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.