57
$\begingroup$

Let's say we want to do regression for simple f = x * y using standard fully-connected deep neural network. The network takes x and y as input, and should learn to output x * y.

I remember that there is research proving that NN with one hidden layer can approximate any function, but I have tried and was unable to approximate even this simple multiplication.

Only log-normalization of the data helped m = x*y => ln(m) = ln(x) + ln(y) because it turns multiplication into addition, but that looks like a cheat. Can NN do this without log-normalization?

The answer seems to be Yes, so the question is more what should the type/configuration/layout of such NN be?

Improve this question
$\endgroup$
0

7 Answers 7

Reset to default
27
$\begingroup$

NN with relu activation function can approximate multiplication when range of inputs is limited. Recall that relu(x) = max(x, 0).

It is enough if NN approximates a square function g(z) = z^2, because x*y = ((x-y)^2 - x^2 - y^2)/(-2). Right-hand side has just linear combinations and squares.

NN can approximate z^2 with a piecewise linear function. For example, on range [0, 2] a combination of x and relu(2(x-1)) is not that bad. Below figure visualises this. No idea if this is useful beyond theory :-) enter image description here

Improve this answer
$\endgroup$
22
$\begingroup$

A big multiplication function gradient forces the net probably almost immediately into some horrifying state where all its hidden nodes have a zero gradient (because of neural network implementation details and limitations). We can use two approaches:

  1. Divide by a constant. We are just dividing everything before the learning and multiply after.

  2. Use log-normalization. It makes multiplication into addition:

    \begin{align} m &= x \cdot y\\ &\Rightarrow \\ \ln(m) &= \ln(x) + \ln(y) \end{align}

Improve this answer
$\endgroup$
20
$\begingroup$

I'm unable to comment due to being a newly active user on StackExchange. But I think this is an important question because it's so friggin simple to understand yet difficult to explain. With respect, I don't think the accepted answer is sufficient. If you think about the core operations of a standard feed-forward NN, with activations of the form s(W*x+b) for some nonlinear activation function s, it's actually not obvious how to "get" multiplication from this even in a composed (multi-layered) network. Scaling (the first bullet in accepted answer) does not seem to address the question at all ... scale by what? The inputs x and y are presumably different for every sample. And taking the log is fine as long as you know that's what you need to do, and take care of the sign issue in preprocessing (since obviously log isn't defined for negative inputs). But this fundamentally doesn't jive with the notion that neural networks can just "learn" (it feels like a cheat as the OP said). I don't think the question should be considered answered until it really is, by someone smarter than me!

Improve this answer
$\endgroup$
1
  • $\begingroup$ The accepted answer indeed does not answer the question, but the answer by @JussiKujala does. $\endgroup$
    – amoeba
    Commented Mar 9, 2023 at 9:58
15
$\begingroup$

A similar question struck me earlier today, and I was surprised I couldn’t find a quick answer. My question was that given NN’s only have summation functions, how could they model multiplicative functions.

This kind of answered it, though it was lengthy explanation. My summary would be that NN’s model the function surface rather than the function itself. Which is obvious, in retrospect…

Improve this answer
$\endgroup$
1
  • $\begingroup$ "The surface" is properly called an "envelope." $\endgroup$
    – Brian Cannard
    Commented Dec 31, 2022 at 23:20
8
$\begingroup$

Traditional neural network consists of linear maps and Lipschitiz activation function. As a composition of Lischitz continuous functions, neural network is also Lipschitz continuous, but multiplication is not Lipschitz continuous. This means that neural network cannot approximate multiplication when one of the x or y goes too large.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Where can I learn more about Lipschitz continuity? So that I can understand why neural networks are and multiplication is not lipschitz continuous. $\endgroup$
    – alvitawa
    Commented Mar 24, 2022 at 13:49
  • 1
    $\begingroup$ @alvitawa A complete understanding of Lipschitz continuity requires a rigorous training in elementary analysis, which is far beyond the scope of most machine learning materials. Lipschitz continuity more or less means having bounded slope. Let's make our life easier. Assume we want to approximate f(x) = x^2 with sigmoid as our activation function so that everything is differentiable. The composition of function of bounded slope is of bounded slope, which indicates that neutral network is of bounded slope. While for f(x) = x^2, you can see the slope becomes larger as x goes to infinity. $\endgroup$
    – fuyutsuki
    Commented Mar 25, 2022 at 7:15
  • 1
    $\begingroup$ Something of bounded slope will always deviate something with unbounded slope when things go to infinity, i.e. neural network cannot even approximate f(x) = x^2. $\endgroup$
    – fuyutsuki
    Commented Mar 25, 2022 at 7:16
3
$\begingroup$

"one hidden layer" does not limit the number of neurons and kinds of activate function used, it still has a large representation space. One simple method to validate the existence of this problem: Train this regress problem with a real neuron network, record each weights and bias, use these parameters plot the predict curve, contrast it with the target function curve. This essay may help.

Improve this answer
$\endgroup$
0
$\begingroup$

Feed-forward REUL, will never be able to learn multiplication exactly or approximately with absolute bounded error, if numbers are unbounded.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Nov 24, 2021 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.