1
$\begingroup$

I am currently learning the very basics of Machine Learning. This e-book looks quite helpful, but I am stuck at the first chapter already.

There, the sigmoid function is defined like normal (equation 3):

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

Next, the output function is this one ($w$ and $x$ are vectors and $b$ is the bias; equation 4):

$$output=\sigma(w·x+b)=\frac{1}{1+exp(-w·x-b)}=\frac{1}{1+exp(-\sum_j{w_jx_j-b})}$$

Suddenly, the author comes up with this approximation term (equation 5):

$$\Delta output\approx \sum_j{\frac{\partial output}{\partial w_j}}+\frac{\partial output}{\partial b}\Delta b$$

Although I do have enough maths knowledge to understand the individual operations (i.e. I know partial derivatives, exp and so on), I do not understand how this last term is derived, especially because it only is an "approximation". Additionally, in contrast to this, what would the exact solution look like?


EDIT: So I tried deriving some of the maths by myself. For now, I ended up with the following:

  • You begin with the vectors $w$ and $x$ and the scalar $b$. The output before is calculated using $\sigma (w·x+b)$.
  • Next, you have the vector $\Delta w$ and the scalar $\Delta b$. Of course these are defined as the changes in $w$ and $b$, so the output after is calculated using $\sigma ((w+\Delta w)·x+(b+\Delta b))$.
  • Apart from that, the output function has very easy partial derivatives: $\frac{\partial output}{\partial w_j}=(1-output)*output*x_j$ and $\frac{\partial output}{\partial b}=(1-output)*output$.

In fact, this formula for approximation is surprisingly good. I created an Excel document to illustrate this: sigmoid approximation As you can see, the approximation only differs by 0.58% from the exact value.

So, I would like to redefine my questions now:

  • Is the "exact solution" I gave in the second bullet point even correct? And if it is: For me, it looks much simpler. Why not use this one instead of needing to calculate complicated partial derivatives?
  • Why is this approximation so good? And how do you "prove" an approximation at all?
$\endgroup$
  • 1
    $\begingroup$ Can't edit this myself, but it should be $\frac 1 {1+ exp(-w \cdot x - b)}$ $\endgroup$ – Tom Hale Feb 17 '18 at 5:04
0
$\begingroup$

You should distinguish term 'derivative' from 'differential'. In this case differential is used. The smaller delta w i-th or delta b is, the better approximation you get.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.