# Approximation of Δoutput in context of Sigmoid function

I am currently learning the very basics of Machine Learning. This e-book looks quite helpful, but I am stuck at the first chapter already.

There, the sigmoid function is defined like normal (equation 3):

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

Next, the output function is this one ($w$ and $x$ are vectors and $b$ is the bias; equation 4):

$$output=\sigma(w·x+b)=\frac{1}{1+exp(-w·x-b)}=\frac{1}{1+exp(-\sum_j{w_jx_j-b})}$$

Suddenly, the author comes up with this approximation term (equation 5):

$$\Delta output\approx \sum_j{\frac{\partial output}{\partial w_j}}+\frac{\partial output}{\partial b}\Delta b$$

Although I do have enough maths knowledge to understand the individual operations (i.e. I know partial derivatives, exp and so on), I do not understand how this last term is derived, especially because it only is an "approximation". Additionally, in contrast to this, what would the exact solution look like?

EDIT: So I tried deriving some of the maths by myself. For now, I ended up with the following:

• You begin with the vectors $w$ and $x$ and the scalar $b$. The output before is calculated using $\sigma (w·x+b)$.
• Next, you have the vector $\Delta w$ and the scalar $\Delta b$. Of course these are defined as the changes in $w$ and $b$, so the output after is calculated using $\sigma ((w+\Delta w)·x+(b+\Delta b))$.
• Apart from that, the output function has very easy partial derivatives: $\frac{\partial output}{\partial w_j}=(1-output)*output*x_j$ and $\frac{\partial output}{\partial b}=(1-output)*output$.

In fact, this formula for approximation is surprisingly good. I created an Excel document to illustrate this: As you can see, the approximation only differs by 0.58% from the exact value.

So, I would like to redefine my questions now:

• Is the "exact solution" I gave in the second bullet point even correct? And if it is: For me, it looks much simpler. Why not use this one instead of needing to calculate complicated partial derivatives?
• Why is this approximation so good? And how do you "prove" an approximation at all?
• Can't edit this myself, but it should be $\frac 1 {1+ exp(-w \cdot x - b)}$ – Tom Hale Feb 17 '18 at 5:04