I implemented a simple feed-forward neural network with biases trained with backpropagation and stochastic gradient decent. What surprises me is how differently is behaves on two datasets I generated.
# Product dataset inputs = np.random.rand(100000, 10) targets = np.column_stack([np.prod(inputs, axis=1)]) # Sum dataset inputs = np.random.rand(100000, 10) targets = np.column_stack([np.sum(inputs, axis=1)])
For the products, the networks quickly converges to an error close zero while it doesn't make much progress on the sums. Note that in the charts, the error on the current batch is plotted rather than evaluating the whole test set every time.
I played with parameters like learning rate, batch size and weight initialization. Tweaking the parameters doesn't seem to have a big impact on the relative performance on the two datasets. This is the configuration I used to produce the charts:
Layer sizes: 10, 15, 15, 1 Activation function: Sigmoid Loss function: Sum of squared errors Initial weights: Gaussian with scale 0.1 Learning rate: 1e-3 Batch size: 100 Rounds on the dataset: 5
Is the problem of finding sums of inputs conceptually harder than finding products for multi-layer perceptrons? My intuition is the other way around: Trivial weights can just pass activations from layer to layer unchanged and they get summed up for the output neuron. For multiplication, I don't see a trivial weight pattern. On the other hand, I don't know if my implementation is correct.