Understanding why deep and cheap learning works so well. Max Tegmark paper

I am reading a paper by Henry Lin and Max Tegmark entitled why does deep and cheap learning work so well.

On the fourth page of the paper they show that it is possible to create a neural network that represents multiplication of two numbers arbitrarily well with a single hidden layer of dimension 4 and input of size 2. I am finding it difficult to replicate the results.

Just going off of the picture that he provides for the multiplication gate it appears that he has equal weights across all of the nodes just alternating signs. Because of this equal weights and sign alternation the output would always be 0. So I am clearly missing something.

The point that perhaps that I am missing is that the neural network takes the form f = A2*sigma*A1 where the As are Affine transformations with an additional bias of the form Ay = Wy + b.

In equation 10,11 of the paper is where they make their conclusions. Theorem: Let f be a neural network of the form $\ f = A_2*\sigma*A_1$

Equation 10 $$\ \sigma(u) \approx \sigma_0 + \sigma_1*u + \sigma_2*u^2/2 + O(u^3)$$

They say that 10 then implies

$\ m(u,v) = (\sigma(u + v) + \sigma(-u - v) - \sigma(u - v) - \sigma(-u + v))/4\sigma_2 = uv*(1 + O(u^2 + v^2))$

I do not end up with the same results when I try using the taylor series expansion. I have tried using $\lambda = 1$ and that has given me the closest result of $\ m(u,v) = 4*u*v*\mu + O((u - v)^3)$

Any insight on the correct direction to take or perhaps any place where the paper is explained a bit more explicitly would be greatly appreciated. Thanks.

Link to the paper Max Tegmark paper

• Can you paste in the section[s] & the picture in question? We don't want someone to have to read the paper just to answer your question. Can you provide a complete citation, in case the link goes dead? – gung Nov 6 '16 at 19:54
• @gung thank you gung I have tried to make it more clear what I am asking. – David Sewell Nov 8 '16 at 21:20

The answer is given in equations 10 and 11 in the linked paper. Basically you take the Taylor expansion of $\sigma(x)\approx\sigma_0+\sigma_1x+\frac{\sigma_2x^2}{2}$ and plug that into the equation formed by the neural net shown in the figure, for $\lambda=1$: $\frac{\sigma(u+v)+\sigma(-u-v)-\sigma(u-v)-\sigma(-u+v)}{4\sigma_2}$ and you get the desired multiplication $uv$. The Taylor expansion is only valid for small parameters, so you need to scale them apropriately by choosing a small $\lambda$ (in the paper they say you need a large $\lambda$, so I might be missing something, but you get the idea).
• thank your for the response. I attempted to solve using the taylor expansion with $\lambda=1$ and I ended with the result $\ m(u,v) = 4*u*v*\mu + O((u-v)^3)$ which is close but not what they got. It does not seem that they use $\lambda=1$ also. I still seem to be missing something probably obvious – David Sewell Nov 8 '16 at 20:38
• But mu is also a free parameter, so if you choose $\mu$ to be 1/4, you are there, correct? – phreeza Nov 9 '16 at 11:29
• close they end up with $\ u*v*(1 + O(u^2 + v^2))$ I should have included that before. I edited my question to include it now – David Sewell Nov 9 '16 at 15:32