# My neural network is good learning products but fails for sums

I implemented a simple feed-forward neural network with biases trained with backpropagation and stochastic gradient decent. What surprises me is how differently is behaves on two datasets I generated.

# Product dataset
inputs = np.random.rand(100000, 10)
targets = np.column_stack([np.prod(inputs, axis=1)])

# Sum dataset
inputs = np.random.rand(100000, 10)
targets = np.column_stack([np.sum(inputs, axis=1)])


For the products, the networks quickly converges to an error close zero while it doesn't make much progress on the sums. Note that in the charts, the error on the current batch is plotted rather than evaluating the whole test set every time.

I played with parameters like learning rate, batch size and weight initialization. Tweaking the parameters doesn't seem to have a big impact on the relative performance on the two datasets. This is the configuration I used to produce the charts:

Layer sizes:           10, 15, 15, 1
Activation function:   Sigmoid
Loss function:         Sum of squared errors
Initial weights:       Gaussian with scale 0.1
Learning rate:         1e-3
Batch size:            100
Rounds on the dataset: 5


Is the problem of finding sums of inputs conceptually harder than finding products for multi-layer perceptrons? My intuition is the other way around: Trivial weights can just pass activations from layer to layer unchanged and they get summed up for the output neuron. For multiplication, I don't see a trivial weight pattern. On the other hand, I don't know if my implementation is correct.

• Are you sure your gradient calculation is correct? – Mark L. Stone Sep 30 '15 at 19:32
• What types of nodes are being used internally? – jlimahaverford Sep 30 '15 at 19:38
• @jlimahaverford Sigmoid. Added that to the description. – danijar Sep 30 '15 at 19:42
• @MarkL.Stone Very possible. I couldn't find anything tough. Here is my code. – danijar Sep 30 '15 at 19:49
• With the sigmoid activation units your idea that "Trivial weights can just pass activations from layer to layer unchanged" is not possible. This does not explain why products is doing better than sums though, in my mind. – jlimahaverford Sep 30 '15 at 20:52