Targets of 0.1/0.9 instead of 0/1 in neural networks and other classification algorithms Rumelhart, Hinton and Williams (PDF) wrote in 1986 in the context of training a neural network (page 12):

One other feature of this activation function should be noted. The
  system can not actually reach its extreme values of 1 or 0 without
  infinitely large weights. Therefore, in a practical learning situation
  in which the desired outputs are binary {0, 1}, the system can never
  actually achieve these values. Therefore, we typically use the values
  of 0.1 and 0.9 as the targets, even though we will talk as if values
  of {0, 1} are sought.

I haven't seen this advice in any more recent paper, nor in any piece of code implementing a neural network. My questions:


*

*Is this advice still valid, or was it disproved as ineffective at some point?

*Was this advice used (historically) in other algorithms, e.g. logistic regression?
 A: Coding the targets as $\{0.1, 0.9\}$ is an example of label smoothing. It's one of the tricks highlighted in this review and it appears to originate in "Rethinking the Inception Architecture for Computer Vision" by Christian Szegedy et al. as a certain kind of regularization. It's used as a regularization strategy to discourage a neural network from giving over-confident predictions in the outcome, because the predicted probabilities cannot be improved by becoming arbitrarily close to 1 for the true class; instead, the lowest loss value occurs at 0.9 for the true class.
The standard approach is to not use label smoothing.  When we estimate a logistic regression (a 0-hidden layer neural network with sigmoid activations on the last layer), we never do this; the problem is left "as is" for Newton's Method to solve. Provided that the usual conditions are satisfied (design matrix is full-rank, perfect separation is not present), we have no problem estimating the regression coefficients. Likewise, neural networks trained on binary problems proceed in the same way! Nonlinearities and multiple layers make the optimization more challenging, but the core concept is the same.
However, neural networks are more flexible models, and are able to find more complex relationships compared to logistic regression. This means that they are also more prone to giving uncalibrated outcomes (the predicted probabilities do not strongly correspond to the true probabilities), a trend that is remarked upon in  "Your classifier is secretly an energy-based model, and you should treat it like one" by Will Grathwohl, et al. (which also has citations to  other papers making similar observaitons).
A: I suspect the reasons they suggested this approach was that the magnitudes of the gradient descent steps in back-propagation are proportional to the derivative of the activation function. If you use a logistic activation function then the derivative is numerically zero before the output reaches 1 or 0.  As a result the optimization can never absolutely converge as the loss function has a very long trough in weight-space with a very shallow slope along the bottom.  Using these modified targets backpropagation is able to converge to a more definite minimum.  I used this technique a bit back in the very early 90s‡, and it doesn't really give much of a benefit in practice, especially if the modification to the targets is as large as that!
A more recent usage of this trick can be found in Platt Scaling, which is used to get estimates of the probability of class membership from support vector machines†.  In Platt Scaling, a logistic regression model is fitted to the (leave-one-out) scores outputted by the SVM, but with modified targets (as @Sycorax explains +1) as a regularisation method to avoid over-fitting.  However Platt adopts a much less heuristic (and less drastic) approach, and sets the targets to be
$y_+ = \frac{N_+ + 1}{N + 2} \qquad \mathrm{and} \qquad  y_- = \frac{N_- + 1}{N + 2}$
which is effectively a Bayesian regularisation based on the Laplace correction.
† Really - don't do this, if you want probabilities (and for most practical applications, you will), use a proper probabilistic classifier, such as kernel logistic regression or Gaussian process classifiers (if you like to be Bayesian).
‡ The PDP books were where I first learned about neural networks! (my copy is from 1989 ;o)
