6
$\begingroup$

Imagine a training set with 10 input features. The first two are mass (M) and accelertion (A) and the rest are all useless or irrelevant. The output is Force (F) which of course is equal to M*A.

It seems to me that an ANN architecture has no easy way to just multiply these two features together and give the answer.

If the input vector was [1,1,0,0,0,0,0,0,0,0] and the weights were all initialized to 1, then you would get (1*M) + (1*A). This doesn't provide the necessary straightforward multiplication.

The ANN will obviously figure out the mapping somehow, but will it always do so in a very over-complicated way?

Is this a limitation in some respects or am I misunderstanding something important about ANN's?

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Modeling analytical functions are popularly called meta modeling! $\endgroup$ – Andreas Storvik Strauman Apr 15 '18 at 18:54
7
$\begingroup$

Yes, the network will probably approximate some sort of multiplication, but it is unlikely to generalize outside the range of inputs you train it on.

You may have more luck learning and generalizing the rule by using a quadratic neuron, which is capable of multiplying inputs. RNTNs introduced here do something to that effect.

I suggest a neuron which does something like $f(x) = a(x^TWx + b^Tx + c)$, where $a$ is the activation function (possibly linear).

$\endgroup$
  • $\begingroup$ You seem to be multiplying x_transpose * W by the full vector x. Don't we need to pass a scalar to a()? $\endgroup$ – COOLBEANS Apr 15 '18 at 17:49
  • $\begingroup$ Yes, $x$ is the output of the previous layer as a vector, and $f$ is the output of a single neuron in the next layer (so each neuron has its own $W$, $b$, and maybe $c$). You can think of $a$ as the vectorized form of whatever activation you want to use. $\endgroup$ – shimao Apr 15 '18 at 17:51
  • 1
    $\begingroup$ @COOLBEANS $x^T W x$ is a scalar, since it's a $(1 \times n)(n \times n)(n \times 1)$ matrix product. $\endgroup$ – David Zhang Apr 16 '18 at 0:00
  • $\begingroup$ If the activation function is linear, doesn't that just make this a glorified logistic regressor? $\endgroup$ – coldspeed Apr 16 '18 at 2:30
  • $\begingroup$ @cᴏʟᴅsᴘᴇᴇᴅ yes, with second order interactions between the inputs, which is all that is necessary in this case. To learn more complex non-linear interactions (such as in the RNTN paper) it makes sense to use a non-linear interaction. $\endgroup$ – shimao Apr 16 '18 at 2:36
8
$\begingroup$

As Shimao said, a NNet will be able to learn some sigmoid based approximation of multiplication, but it will likely fail for new values that fall outside of the range of the training set.

Remember that a 3 layer feedforward neural net is a universal approximator, given a sufficient number of neurons. So you might end up having to resort to having one neuron per training example, in which case your NNet isn't really "learning" anything, it is just acting as a lookup table.

Another approach to your problem is to perform a log transform:

$F = ma \rightarrow log(F) = log(m) + log(a)$

A NNet can easily learn the second function, and then reverse transfrom the output.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.