In lecture 9.1 of Geoffrey Hinton's course Neural Networks for Machine Learning, he offers a justification for why "early stopping" as a method to improve generalisation works.
He states that using a network with logistic hidden units and small weights, the total inputs to these hidden units will be close to 0 and therefore in the middle of the linear range of the sigmoid and will thus behave like a linear network (having a lower capacity / expressive power).
Where is the linear region of the sigmoid function? I would have thought that it would be in the middle of the curve, around 0 on the horizontal axis. (I've highlighted the area in question in the image below)
However, since he mentions that the total inputs being close to 0 when the sigmoid behaves in a linear fashion it sounds like he's referring to the curved taper to 0 as the output (in the bottom left of the picture).
Where is the linear region of the sigmoid?
2*sigmoid(t) - 1
, which scales and re-centers to that the curve passes through the origin and asymptotes to minus one and one. $\endgroup$