1
$\begingroup$

I implemented a simple feed-forward neural network with biases trained with backpropagation and stochastic gradient decent. What surprises me is how differently is behaves on two datasets I generated.

# Product dataset
inputs = np.random.rand(100000, 10)
targets = np.column_stack([np.prod(inputs, axis=1)])

# Sum dataset
inputs = np.random.rand(100000, 10)
targets = np.column_stack([np.sum(inputs, axis=1)])

For the products, the networks quickly converges to an error close zero while it doesn't make much progress on the sums. Note that in the charts, the error on the current batch is plotted rather than evaluating the whole test set every time.

errors on product dataset

errors on sum dataset

I played with parameters like learning rate, batch size and weight initialization. Tweaking the parameters doesn't seem to have a big impact on the relative performance on the two datasets. This is the configuration I used to produce the charts:

Layer sizes:           10, 15, 15, 1
Activation function:   Sigmoid
Loss function:         Sum of squared errors
Initial weights:       Gaussian with scale 0.1
Learning rate:         1e-3
Batch size:            100
Rounds on the dataset: 5

Is the problem of finding sums of inputs conceptually harder than finding products for multi-layer perceptrons? My intuition is the other way around: Trivial weights can just pass activations from layer to layer unchanged and they get summed up for the output neuron. For multiplication, I don't see a trivial weight pattern. On the other hand, I don't know if my implementation is correct.

$\endgroup$
8
  • $\begingroup$ Are you sure your gradient calculation is correct? $\endgroup$ Sep 30, 2015 at 19:32
  • $\begingroup$ What types of nodes are being used internally? $\endgroup$ Sep 30, 2015 at 19:38
  • $\begingroup$ @jlimahaverford Sigmoid. Added that to the description. $\endgroup$
    – danijar
    Sep 30, 2015 at 19:42
  • $\begingroup$ @MarkL.Stone Very possible. I couldn't find anything tough. Here is my code. $\endgroup$
    – danijar
    Sep 30, 2015 at 19:49
  • $\begingroup$ With the sigmoid activation units your idea that "Trivial weights can just pass activations from layer to layer unchanged" is not possible. This does not explain why products is doing better than sums though, in my mind. $\endgroup$ Sep 30, 2015 at 20:52

1 Answer 1

3
$\begingroup$

If you are using a sigmoid units everywhere and 10 inputs that are uniformly distributed between [0,1] then, while their product will be in [0,1], their sum will most likely not be, guaranteeing a certain amount of error.

P.S. - To be honest, I'm confused as to why the error is as low as it is.

$\endgroup$
6
  • $\begingroup$ Always normalize your data. Lesson learned :) Thanks for pointing me in that direction. $\endgroup$
    – danijar
    Sep 30, 2015 at 21:57
  • $\begingroup$ Can you also try using linear units? $\endgroup$ Sep 30, 2015 at 22:17
  • $\begingroup$ For ReLU less and less neurons get activated after some training and for the identity activation functions, gradients get huge and overflow after some training. Both don't work very well. Suggestions are welcome. (If I cancel training before the gradients get huge, the identity activation functions get a very small error.) $\endgroup$
    – danijar
    Oct 1, 2015 at 2:09
  • $\begingroup$ That seems absurd. I will be thinking about that a lot! $\endgroup$ Oct 1, 2015 at 2:17
  • $\begingroup$ I got the formula for backpropagation wrong in a way that it didn't affect learning with sigmoids. However, identity and ReLU don't learn anything. I tried very small learning rates but it doesn't help. Do you have some advice on how to train with those activation functions? Can I use them as for the output layer as well? $\endgroup$
    – danijar
    Oct 1, 2015 at 4:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.