I'm teaching myself about neural networks to implement a simple one in C++ and have come to a roadblock. I am just starting to understand the math behind gradient descent and some of the other topics involved however what I don't understand is the exact implementation.

Assume I have a network with a depth of 3, input/hidden/output, after my hidden layer calculates the sigmoid from the weigted/biased value sent to each neuron does a hidden neuron pass on the sigmoid value to the next dendrite in the chain or does it convert it to a binary output? My assumption is that it converts it to binary but I keep hitting walls and questioning myself.

If both are possible which one is more efficient in terms of training speed? Forgive me if I have worded this poorly, like I stated I am still learning.

It passes on the result of the sigmoid function and does not binarize the output. The reason for this is that the neural network must (usually) be differentiable for the backpropagation training algorithm to work. The binarization function (rounding) is not differentiable.

And a small note about depth. When talking about neural networks, the term depth does not include the input layer. It's just the number of hidden layers + the output layer.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.