How can't the Softmax layer never converge using hard targets

Question

Here's a quote from the deep learning book by Ian Goodfellow (page 236):

Maximum likelihood learning with a softmax classifier and hard targets may actually never converge -- the softmax can never predict a probability of exactly 0 or 1...

I have built many DNN models and I used Softmax layer as a classification layer but I actually never noticed that this is possible. Actually, I think it does make sense, but why don't we face that in practice? Do frameworks "terminate" the gradient descent algorithm earlier and handle this issue internally? I studied many books and articles about DNN but this is the first time that I read something about that. Or, is that valid only in some contexts?

Answer 1

The wording never converge may sound a bit too strong, but the actual statement is

... the softmax can never predict a probability of exactly $0$ or exactly $1$, ...

This is certainly true in almost all cases. In this context, a convergence means fitting the training data perfectly and outputting a one-hot vector of probabilities for all inputs $x$. In all other cases, there will be some loss, which basically means the algorithm hasn't converged yet. And this is exactly what happens in practice: usually, the learning stops either when the researches sees no improvements in training or the time limit expires.

By the way, the quote is taken from the chapter on Regularization, and there authors explain that fitting the training data perfectly is a bad idea and injecting noise to the learning process actually improves generalization.

Answer 2

why don't we face that in practice?

The predictions for multiclass classification are done by taking argmax over probability vector, so this is not really an issue.

Do frameworks "terminate" the gradient descent algorithm earlier and handle this issue internally?

In deep learning usually you don't have any convergence guarantee, so most frameworks just assume you specify number of iterations, or tolerance (for example you stop if log-loss changes less than $\epsilon$ between iterations).

There is also the problem with being too confident, but Maxim covered that. For a concrete example you can see this paper.