Let's say that I train a neural network in a classic binary classification setting where all the training data has labels in $\{-1, +1\}$. From my understanding, if I train the network with a log-loss function and softmax output layer, the network outputs will essentially estimate $P(y \mid \mathbf{x})$.

How do I prove this convergence mathematically? In other words, if $h(\mathbf{x};y)$ is the output of the network, how do I show that the minimizing the loss brings $h(\mathbf{x};y)$ closer to $P(y \mid \mathbf{x})$ over all $\mathbf{x}$ over time?


  • 1
    $\begingroup$ It sounds like you are looking for a mathematical proof that the log loss, as a function of the predicted probability, is minimized by the true probability. Is that correct? (This amounts to proving that the log loss is a proper scoring rule.) If so, this is a straightforward optimization exercise with a little calculus Please add the self-study tag & read its wiki. $\endgroup$ Commented Mar 25, 2023 at 7:47
  • $\begingroup$ @StephanKolassa Yes, I think that's exactly what I'm trying to do. Thanks for the insight, I'll give it an attempt myself based on what you said. $\endgroup$ Commented Mar 25, 2023 at 7:55
  • $\begingroup$ Notice that in order to converge to the true probabilities you need to assume that the network has the ability to approximate arbitrary functions (e.g. by relying on universal approximation theorems. A given network with a fixed number of neuron will not have this ability. $\endgroup$
    – J. Delaney
    Commented Mar 25, 2023 at 12:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.