I am reading a paper by Henry Lin and Max Tegmark entitled why does deep and cheap learning work so well.
On the fourth page of the paper they show that it is possible to create a neural network that represents multiplication of two numbers arbitrarily well with a single hidden layer of dimension 4 and input of size 2. I am finding it difficult to replicate the results.
Just going off of the picture that he provides for the multiplication gate it appears that he has equal weights across all of the nodes just alternating signs. Because of this equal weights and sign alternation the output would always be 0. So I am clearly missing something.
The point that perhaps that I am missing is that the neural network takes the form f = A2*sigma*A1 where the As are Affine transformations with an additional bias of the form Ay = Wy + b.
In equation 10,11 of the paper is where they make their conclusions. Theorem: Let f be a neural network of the form $\ f = A_2*\sigma*A_1 $
Equation 10 $$\ \sigma(u) \approx \sigma_0 + \sigma_1*u + \sigma_2*u^2/2 + O(u^3) $$
They say that 10 then implies
$\ m(u,v) = (\sigma(u + v) + \sigma(-u - v) - \sigma(u - v) - \sigma(-u + v))/4\sigma_2 = uv*(1 + O(u^2 + v^2)) $
I do not end up with the same results when I try using the taylor series expansion. I have tried using $ \lambda = 1 $ and that has given me the closest result of $\ m(u,v) = 4*u*v*\mu + O((u - v)^3) $
Any insight on the correct direction to take or perhaps any place where the paper is explained a bit more explicitly would be greatly appreciated. Thanks.
Link to the paper Max Tegmark paper