42
$\begingroup$

Let say we want to do regression for simple f = x * y using standart deep neural network.

I remember that there are reseraches that tells that NN with one hiden layer can apoximate any function, but I have tried and without normalization NN was unable approximate even this simple multiplication. Only log-normalization of data helped m = x*y => ln(m) = ln(x) + ln(y). But that looks like a cheat. Can NN do this without log-normalization? The unswer is obviously(as for me) - yes, so the question is more what should be type/configuration/layout of such NN?

$\endgroup$
0

7 Answers 7

23
$\begingroup$

A big multiplication function gradient forces the net probably almost immediately into some horrifying state where all its hidden nodes have a zero gradient (because of neural network implementation details and limitations). We can use two approaches:

  1. Divide by a constant. We are just dividing everything before the learning and multiply after.

  2. Use log-normalization. It makes multiplication into addition:

    \begin{align} m &= x \cdot y\\ &\Rightarrow \\ \ln(m) &= \ln(x) + \ln(y) \end{align}

$\endgroup$
17
$\begingroup$

NN with relu activation function can approximate multiplication when range of inputs is limited. Recall that relu(x) = max(x, 0).

It is enough if NN approximates a square function g(z) = z^2, because x*y = ((x-y)^2 - x^2 - y^2)/(-2). Right-hand side has just linear combinations and squares.

NN can approximate z^2 with a piecewise linear function. For example, on range [0, 2] a combination of x and relu(2(x-1)) is not that bad. Below figure visualises this. No idea if this is useful beyond theory :-) enter image description here

$\endgroup$
13
$\begingroup$

A similar question struck me earlier today, and I was surprised I couldn’t find a quick answer. My question was that given NN’s only have summation functions, how could they model multiplicative functions.

This kind of answered it, though it was lengthy explanation. My summary would be that NN’s model the function surface rather than the function itself. Which is obvious, in retrospect…

$\endgroup$
12
$\begingroup$

I'm unable to comment due to being a newly active user on StackExchange. But I think this is an important question because it's so friggin simple to understand yet difficult to explain. With respect, I don't think the accepted answer is sufficient. If you think about the core operations of a standard feed-forward NN, with activations of the form s(W*x+b) for some nonlinear activation function s, it's actually not obvious how to "get" multiplication from this even in a composed (multi-layered) network. Scaling (the first bullet in accepted answer) does not seem to address the question at all ... scale by what? The inputs x and y are presumably different for every sample. And taking the log is fine as long as you know that's what you need to do, and take care of the sign issue in preprocessing (since obviously log isn't defined for negative inputs). But this fundamentally doesn't jive with the notion that neural networks can just "learn" (it feels like a cheat as the OP said). I don't think the question should be considered answered until it really is, by someone smarter than me!

$\endgroup$
6
$\begingroup$

Traditional neural network consists of linear maps and Lipschitiz activation function. As a composition of Lischitz continuous functions, neural network is also Lipschitz continuous, but multiplication is not Lipschitz continuous. This means that neural network cannot approximate multiplication when one of the x or y goes too large.

$\endgroup$
3
  • $\begingroup$ Where can I learn more about Lipschitz continuity? So that I can understand why neural networks are and multiplication is not lipschitz continuous. $\endgroup$
    – alvitawa
    Mar 24 at 13:49
  • 1
    $\begingroup$ @alvitawa A complete understanding of Lipschitz continuity requires a rigorous training in elementary analysis, which is far beyond the scope of most machine learning materials. Lipschitz continuity more or less means having bounded slope. Let's make our life easier. Assume we want to approximate f(x) = x^2 with sigmoid as our activation function so that everything is differentiable. The composition of function of bounded slope is of bounded slope, which indicates that neutral network is of bounded slope. While for f(x) = x^2, you can see the slope becomes larger as x goes to infinity. $\endgroup$
    – fuyutsuki
    Mar 25 at 7:15
  • 1
    $\begingroup$ Something of bounded slope will always deviate something with unbounded slope when things go to infinity, i.e. neural network cannot even approximate f(x) = x^2. $\endgroup$
    – fuyutsuki
    Mar 25 at 7:16
3
$\begingroup$

"one hidden layer" does not limit the number of neurons and kinds of activate function used, it still has a large representation space. One simple method to validate the existence of this problem: Train this regress problem with a real neuron network, record each weights and bias, use these parameters plot the predict curve, contrast it with the target function curve. This essay may help.

$\endgroup$
0
$\begingroup$

Feed-forward REUL, will never be able to learn multiplication exactly or approximately with absolute bounded error, if numbers are unbounded.

$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Nov 24, 2021 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.