I don't know what I'm doing wrong. I'm using DL4J, and I took their XorExample class (which trains a feed-forward network to perform xor) and tried to modify it to multiply two inputs between 0 and 1 (500 samples of this; three values: the two inputs, and their product). The weights appeared to just explode (or vanish, it flipped at some point), so I changed to plain addition (of two inputs each between 0 and 0.5). Continued divergence. Then I changed the activation functions from TANH to SIGMOID. Continued divergence. Then from SIGMOID to IDENTITY, figuring that I'd throw it a soft ball and let its activation function do the adding, alone. The weights continued to increase unboundedly as iterations increased. (I'm inferring this from the output increasing unboundedly.) I removed all hidden layers, and all I'm left with (if I'm not misreading DL4J's syntax) is two nodes taking from two inputs, and one output node taking from the two nodes in the first layer. At each step, I've tried varying the iteration count and the learning rate. The weights continue to explode. I am baffled as to why.
My code, as it stands, can be found here: http://pastebin.com/kv6jjGM4
Reading some of the settings straight out of the code, it's using stochastic gradient descent for its optimization algorithm, with negative log likelihood for its loss function. (I admit to only partially understanding what those are.)
Either way, unless one of those two settings entirely reverses something, shouldn't the optimization at least cause the results to stay between 0 and 1, since that's where all outputs are?