5
$\begingroup$

Imagine a training set with 10 input features. The first two are mass (M) and accelertion (A) and the rest are all useless or irrelevant. The output is Force (F) which of course is equal to M*A.

It seems to me that an ANN architecture has no easy way to just multiply these two features together and give the answer.

If the input vector was [1,1,0,0,0,0,0,0,0,0] and the weights were all initialized to 1, then you would get (1*M) + (1*A). This doesn't provide the necessary straightforward multiplication.

The ANN will obviously figure out the mapping somehow, but will it always do so in a very over-complicated way?

Is this a limitation in some respects or am I misunderstanding something important about ANN's?

enter image description here

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Modeling analytical functions are popularly called meta modeling! $\endgroup$
    – Andreas Storvik Strauman
    Commented Apr 15, 2018 at 18:54

2 Answers 2

Reset to default
7
$\begingroup$

Yes, the network will probably approximate some sort of multiplication, but it is unlikely to generalize outside the range of inputs you train it on.

You may have more luck learning and generalizing the rule by using a quadratic neuron, which is capable of multiplying inputs. RNTNs introduced here do something to that effect.

I suggest a neuron which does something like $f(x) = a(x^TWx + b^Tx + c)$, where $a$ is the activation function (possibly linear).

Improve this answer
$\endgroup$
5
  • $\begingroup$ You seem to be multiplying x_transpose * W by the full vector x. Don't we need to pass a scalar to a()? $\endgroup$
    – COOLBEANS
    Commented Apr 15, 2018 at 17:49
  • $\begingroup$ Yes, $x$ is the output of the previous layer as a vector, and $f$ is the output of a single neuron in the next layer (so each neuron has its own $W$, $b$, and maybe $c$). You can think of $a$ as the vectorized form of whatever activation you want to use. $\endgroup$
    – shimao
    Commented Apr 15, 2018 at 17:51
  • 1
    $\begingroup$ @COOLBEANS $x^T W x$ is a scalar, since it's a $(1 \times n)(n \times n)(n \times 1)$ matrix product. $\endgroup$
    – David Zhang
    Commented Apr 16, 2018 at 0:00
  • $\begingroup$ If the activation function is linear, doesn't that just make this a glorified logistic regressor? $\endgroup$
    – coldspeed
    Commented Apr 16, 2018 at 2:30
  • $\begingroup$ @cᴏʟᴅsᴘᴇᴇᴅ yes, with second order interactions between the inputs, which is all that is necessary in this case. To learn more complex non-linear interactions (such as in the RNTN paper) it makes sense to use a non-linear interaction. $\endgroup$
    – shimao
    Commented Apr 16, 2018 at 2:36
7
$\begingroup$

As Shimao said, a NNet will be able to learn some sigmoid based approximation of multiplication, but it will likely fail for new values that fall outside of the range of the training set.

Remember that a 3 layer feedforward neural net is a universal approximator, given a sufficient number of neurons. So you might end up having to resort to having one neuron per training example, in which case your NNet isn't really "learning" anything, it is just acting as a lookup table.

Another approach to your problem is to perform a log transform:

$F = ma \rightarrow log(F) = log(m) + log(a)$

A NNet can easily learn the second function, and then reverse transfrom the output.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.