Logistic functions in ANN. Biologically inspired? Was the introduction of logistic/sigmoid activation functions in artificial networks justified by biological evidence, practical convenience or both? What are some good references on this topic?
 A: The logistic function is a convenience.
The modern inspiration for artificial neural networks comes from A logical calculus of the ideas immanent in nervous activity. The proposed neuron is based on a realistic binary neuron, the threshold logic unit. This is the Heaviside function.
$ H(x) = \left\{
     \begin{array}{lr}
       1 ,  w  x +b > t\\
       0 , \text{otherwise}\\
     \end{array}
   \right.$
Assembling nodes with this activation function in a many-to-one mapping is called a perceptron. This is one of the first ANN algorithms that saw wide-spread use. The main reason was because a simple training algorithm was discovered. The update rule is similar to gradient descent of ordinary least squares.
$w(t+1) = w(t) + \alpha (y_{true} - y_{pred})x$
One of several reasons that the perceptron fell out of favor is that the perceptron is not a universal function approximator. The perceptron can only learn a linear separation boundary.
The XOR function for instance cannot be learned by the perceptron.
$ X = \left[ \begin{array}{ccc}
0 & 0\\
1 & 0 \\
0 & 1 \\
1 & 1 \end{array} \right]$
$ Y = \left[ \begin{array}{ccc}
1\\
0\\
0\\
1\end{array} \right]$
When MLPs came into the picture, so did backprop. MLPs are universal function approximators. We could use binary neurons with MLPs but the error is high, it takes longer to converge and requires more neurons.
The recent trend is towards Rectified Linear Units regularized with Dropout. Our modern understanding of real neural networks is the stochastic binary neuron, which is a probabilistically activated Heaviside function.
Dropout can be seen as mimicking the regularizing behaviour of the stochastic binary neuron. It can also been seen as a form of ridge regression with a diag(covariance) prior. The deep connection between all three is fascinating.
A: Not particularly. The sigmoid function has interesting mathematical properties. In particular, the sigmoid is the link function for the generalized linear model for binomial response data. Logistic regression as a special case existed before the advent of theory surrounding GLMs. GLMs greatly codify this, however. The relationship between the mean-variance structure and the score equations for the logistic regression model give the log-odds ratio as the natural parameter to a sigmoid-linear mean model.
To understand this, it would help to review the McCullagh Nelder text "Generalized Linear Models", 2nd ed.
