0
$\begingroup$

I don't know what I'm doing wrong. I'm using DL4J, and I took their XorExample class (which trains a feed-forward network to perform xor) and tried to modify it to multiply two inputs between 0 and 1 (500 samples of this; three values: the two inputs, and their product). The weights appeared to just explode (or vanish, it flipped at some point), so I changed to plain addition (of two inputs each between 0 and 0.5). Continued divergence. Then I changed the activation functions from TANH to SIGMOID. Continued divergence. Then from SIGMOID to IDENTITY, figuring that I'd throw it a soft ball and let its activation function do the adding, alone. The weights continued to increase unboundedly as iterations increased. (I'm inferring this from the output increasing unboundedly.) I removed all hidden layers, and all I'm left with (if I'm not misreading DL4J's syntax) is two nodes taking from two inputs, and one output node taking from the two nodes in the first layer. At each step, I've tried varying the iteration count and the learning rate. The weights continue to explode. I am baffled as to why.

My code, as it stands, can be found here: http://pastebin.com/kv6jjGM4

Reading some of the settings straight out of the code, it's using stochastic gradient descent for its optimization algorithm, with negative log likelihood for its loss function. (I admit to only partially understanding what those are.)

Either way, unless one of those two settings entirely reverses something, shouldn't the optimization at least cause the results to stay between 0 and 1, since that's where all outputs are?

$\endgroup$

closed as off-topic by Sycorax, kjetil b halvorsen, conjugateprior, Michael Chernick, John Feb 1 '17 at 0:17

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Sycorax, kjetil b halvorsen, John
If this question can be reworded to fit the rules in the help center, please edit the question.

0
$\begingroup$

Turns out the key was the loss function. I still don't understand why, but when I changed (on line 77) NEGATIVELOGLIKELIHOOD to something else (like POISSON, or L2; L2 gave the best results of the ones I tried), it stopped degenerating and got much better results, and now works for multiplication. (And when I added learning decay, the error decreased by several orders of magnitude. I also tinkered with the activation function once it started giving me good results.)

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.