Can deep neural network approximate multiplication function without normalization?

Question

Let say we want to do regression for simple f = x * y using standart deep neural network.

I remember that there are reseraches that tells that NN with one hiden layer can apoximate any function, but I have tried and without normalization NN was unable approximate even this simple multiplication. Only log-normalization of data helped m = x*y => ln(m) = ln(x) + ln(y). But that looks like a cheat. Can NN do this without log-normalization? The unswer is obviously(as for me) - yes, so the question is more what should be type/configuration/layout of such NN?

Answer 1

A big multiplication function gradient forces the net probably almost immediately into some horrifying state where all its hidden nodes have a zero gradient (because of neural network implementation details and limitations). We can use two approaches:

1. Divide by a constant. We are just dividing everything before the learning and multiply after.

2. Use log-normalization. It makes multiplication into addition:

   \begin{align} m &= x \cdot y\\ &\Rightarrow \\ \ln(m) &= \ln(x) + \ln(y) \end{align}

Answer 2

NN with relu activation function can approximate multiplication when range of inputs is limited. Recall that relu(x) = max(x, 0).

It is enough if NN approximates a square function g(z) = z^2, because x*y = ((x-y)^2 - x^2 - y^2)/(-2). Right-hand side has just linear combinations and squares.

NN can approximate z^2 with a piecewise linear function. For example, on range [0, 2] a combination of x and relu(2(x-1)) is not that bad. Below figure visualises this. No idea if this is useful beyond theory :-)

Answer 3

A similar question struck me earlier today, and I was surprised I couldn’t find a quick answer. My question was that given NN’s only have summation functions, how could they model multiplicative functions.

This kind of answered it, though it was lengthy explanation. My summary would be that NN’s model the function surface rather than the function itself. Which is obvious, in retrospect…

Answer 4

I'm unable to comment due to being a newly active user on StackExchange. But I think this is an important question because it's so friggin simple to understand yet difficult to explain. With respect, I don't think the accepted answer is sufficient. If you think about the core operations of a standard feed-forward NN, with activations of the form s(W*x+b) for some nonlinear activation function s, it's actually not obvious how to "get" multiplication from this even in a composed (multi-layered) network. Scaling (the first bullet in accepted answer) does not seem to address the question at all ... scale by what? The inputs x and y are presumably different for every sample. And taking the log is fine as long as you know that's what you need to do, and take care of the sign issue in preprocessing (since obviously log isn't defined for negative inputs). But this fundamentally doesn't jive with the notion that neural networks can just "learn" (it feels like a cheat as the OP said). I don't think the question should be considered answered until it really is, by someone smarter than me!

Answer 5

"one hidden layer" does not limit the number of neurons and kinds of activate function used, it still has a large representation space. One simple method to validate the existence of this problem: Train this regress problem with a real neuron network, record each weights and bias, use these parameters plot the predict curve, contrast it with the target function curve. This essay may help.

Answer 6

Traditional neural network consists of linear maps and Lipschitiz activation function. As a composition of Lischitz continuous functions, neural network is also Lipschitz continuous, but multiplication is not Lipschitz continuous. This means that neural network cannot approximate multiplication when one of the x or y goes too large.