I implemented a simple feed-forward neural network with biases trained with backpropagation and stochastic gradient decent. What surprises me is how differently is behaves on two datasets I generated.

# Product dataset
inputs = np.random.rand(100000, 10)
targets = np.column_stack([np.prod(inputs, axis=1)])

# Sum dataset
inputs = np.random.rand(100000, 10)
targets = np.column_stack([np.sum(inputs, axis=1)])

For the products, the networks quickly converges to an error close zero while it doesn't make much progress on the sums. Note that in the charts, the error on the current batch is plotted rather than evaluating the whole test set every time.

errors on product dataset

errors on sum dataset

I played with parameters like learning rate, batch size and weight initialization. Tweaking the parameters doesn't seem to have a big impact on the relative performance on the two datasets. This is the configuration I used to produce the charts:

Layer sizes:           10, 15, 15, 1
Activation function:   Sigmoid
Loss function:         Sum of squared errors
Initial weights:       Gaussian with scale 0.1
Learning rate:         1e-3
Batch size:            100
Rounds on the dataset: 5

Is the problem of finding sums of inputs conceptually harder than finding products for multi-layer perceptrons? My intuition is the other way around: Trivial weights can just pass activations from layer to layer unchanged and they get summed up for the output neuron. For multiplication, I don't see a trivial weight pattern. On the other hand, I don't know if my implementation is correct.

  • $\begingroup$ Are you sure your gradient calculation is correct? $\endgroup$ – Mark L. Stone Sep 30 '15 at 19:32
  • $\begingroup$ What types of nodes are being used internally? $\endgroup$ – jlimahaverford Sep 30 '15 at 19:38
  • $\begingroup$ @jlimahaverford Sigmoid. Added that to the description. $\endgroup$ – danijar Sep 30 '15 at 19:42
  • $\begingroup$ @MarkL.Stone Very possible. I couldn't find anything tough. Here is my code. $\endgroup$ – danijar Sep 30 '15 at 19:49
  • $\begingroup$ With the sigmoid activation units your idea that "Trivial weights can just pass activations from layer to layer unchanged" is not possible. This does not explain why products is doing better than sums though, in my mind. $\endgroup$ – jlimahaverford Sep 30 '15 at 20:52

If you are using a sigmoid units everywhere and 10 inputs that are uniformly distributed between [0,1] then, while their product will be in [0,1], their sum will most likely not be, guaranteeing a certain amount of error.

P.S. - To be honest, I'm confused as to why the error is as low as it is.

  • $\begingroup$ Always normalize your data. Lesson learned :) Thanks for pointing me in that direction. $\endgroup$ – danijar Sep 30 '15 at 21:57
  • $\begingroup$ Can you also try using linear units? $\endgroup$ – jlimahaverford Sep 30 '15 at 22:17
  • $\begingroup$ For ReLU less and less neurons get activated after some training and for the identity activation functions, gradients get huge and overflow after some training. Both don't work very well. Suggestions are welcome. (If I cancel training before the gradients get huge, the identity activation functions get a very small error.) $\endgroup$ – danijar Oct 1 '15 at 2:09
  • $\begingroup$ That seems absurd. I will be thinking about that a lot! $\endgroup$ – jlimahaverford Oct 1 '15 at 2:17
  • $\begingroup$ I got the formula for backpropagation wrong in a way that it didn't affect learning with sigmoids. However, identity and ReLU don't learn anything. I tried very small learning rates but it doesn't help. Do you have some advice on how to train with those activation functions? Can I use them as for the output layer as well? $\endgroup$ – danijar Oct 1 '15 at 4:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.