0
$\begingroup$

Why does Backpropagation Algorithm backpropagate a value back on a neuron with activation zero which can't have an influence on the error ? I assume binary activations of the neurons.

float[] feedbackError(bool[] inputs, float[] errors, bool[] output){

float[] feedbackError = new float[inputs.length];
for(int i = 0; i < inputs.length; i++){
  float errorSum = 0.0;
  for(int j = 0; j < this.weights[i].length; j++){
        errorSum           += errors[j] * weights[i][j] * output[i] ; !!! This is the line I'm talking about and I added the output node which has to be corrected as multiplicator !!!
        this.weights[i][j] += inputs[i] * errors[j] * 0.1 ;
  }
  feedbackError[i] = errorSum;
}
//Process Bias (? doesn't count to feedbackError:(!)

int i = weights.length-1;
float errorSum = 0.0;
for(int j = 0; j < this.weights[i].length; j++){
      this.weights[i][j] += 1.0       * errors[j] * 0.1;
}

return feedbackError;

}

I suggest also another possible approach in my repository: Another potential candidate for training several layers in one displayed matrix (Be careful, it's very Alpha and does not converge smoothly but it's very easy and efficient in paralelization)

$\endgroup$
0
$\begingroup$

Think of it as a trial and error scenario. If the error is backpropagated, the weights will change through gradient descent. In the next forward pass, the new weights with the neuron's new input might help the neuron activate (if the neuron's output is above a threshold). If the activated neuron helps reduce the loss, the previously dead neuron has now "learned" some relationship between the input and output through its updated weights and activation.

On the other hand, if the neuron activation led to an increase in loss, backpropagation algorithm will trace back the loss (or error) and update the weights through gradient descent to ensure the neuron is dead in future passes. Practically, this scenario is very rare, since every neuron should learn something about a task. If you have dead neurons in your model, then your model is said to be performing inefficiently.

This trial and error (or to and fro) is called learning or training a model.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I can't understand why a node that has no influence to the result should be considered in the equation. It' just too irrational. $\endgroup$ – schwenk Mar 13 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.