Neural network hidden layer output

Question

I'm teaching myself about neural networks to implement a simple one in C++ and have come to a roadblock. I am just starting to understand the math behind gradient descent and some of the other topics involved however what I don't understand is the exact implementation.

Assume I have a network with a depth of 3, input/hidden/output, after my hidden layer calculates the sigmoid from the weigted/biased value sent to each neuron does a hidden neuron pass on the sigmoid value to the next dendrite in the chain or does it convert it to a binary output? My assumption is that it converts it to binary but I keep hitting walls and questioning myself.

If both are possible which one is more efficient in terms of training speed? Forgive me if I have worded this poorly, like I stated I am still learning.

Answer 1

It passes on the result of the sigmoid function and does not binarize the output. The reason for this is that the neural network must (usually) be differentiable for the backpropagation training algorithm to work. The binarization function (rounding) is not differentiable.

And a small note about depth. When talking about neural networks, the term depth does not include the input layer. It's just the number of hidden layers + the output layer.