What is the linear region of the sigmoid function?

Question

In lecture 9.1 of Geoffrey Hinton's course Neural Networks for Machine Learning, he offers a justification for why "early stopping" as a method to improve generalisation works.

He states that using a network with logistic hidden units and small weights, the total inputs to these hidden units will be close to 0 and therefore in the middle of the linear range of the sigmoid and will thus behave like a linear network (having a lower capacity / expressive power).

Where is the linear region of the sigmoid function? I would have thought that it would be in the middle of the curve, around 0 on the horizontal axis. (I've highlighted the area in question in the image below)

However, since he mentions that the total inputs being close to 0 when the sigmoid behaves in a linear fashion it sounds like he's referring to the curved taper to 0 as the output (in the bottom left of the picture).

Where is the linear region of the sigmoid?

Answer 1

However, since he mentions that the total inputs being close to 0 when the >sigmoid behaves in a linear fashion it sounds like he's referring to the curved taper to 0 as the output (in the bottom left of the picture)

The total input is 0 when it is in the region you have circled. The bottom left corner refers to an input of -inf. Remeber x axis is input, y axis is the output.

Answer 2

The definition of the "linear region" is not clear to me. Because the function is a non-linear function. For a "small" region, it can be approximated by a linear function. Note, it is an approximation, but not really linear.

Any function can be approximated with a linear function with certain "accuracy", how accurate do you want and how do you quantify the accuracy you want? These two questions are the key to answer what is the region of $x$.