6
$\begingroup$

While building a neural network with one hidden layer, the question arose whether or not to update the biases during backpropagation. I'm basically trying to save up on memory, so my question was and is how big a difference it would make if I updated only the weights compared to the weights and the biases. With the former, I wouldn't have to save any other bias value than the value of $1$ which I set as standard. Will it have trouble learning then? If so, then why are the weights updates insufficient for training it?

EDIT for clarity: I'm talking about the backpropagation formula

$\Delta W= -ηδ_l O_{(l-1)}$

$\Delta \theta=-ηδ_l$

Where $\Delta W$ is the difference (vector) of weights, $\Delta \theta$ is the difference (vector) of biases, $η$ is the learning rate, $O$ is the output (vector) of the layer (here $l-1$), and $\delta_l$ is the calculated error increment (vector) of layer $l$. What if you just don't use $\Delta \theta$ in backpropagation and leave the biases at $1$?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

If you try to leave the biases fixed at any value, then each neuron will try to use its "overall input activation" as a kind of bias (by having a small weight to all of its inputs). This makes your learning less stable than just having a bias. If you try to fight this desire with regularization, it may not be possible for the network to encode a good solution to your problem.

How much memory are you really saving?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Well, we're talking about approx. 10^6 nodes, in that dimension. Your point seems legit. So, taken your answer and @runDOSrun 's answer together, one could say that for a stable learning function, you would either need a bias neuron with a fixed value and weights that can be updated, or biases for each neuron that can be updated as well, right? $\endgroup$
    – cirko
    Commented Jul 5, 2015 at 10:53
  • 1
    $\begingroup$ @cirko: yes, that's the same thing. Also, a million biases is only a few megabytes of memory. $\endgroup$
    – Neil G
    Commented Jul 5, 2015 at 15:11
5
$\begingroup$

The idea is to learn the bias weights but have the activation fixed at 1. Anything else would make it an additional ordinary unit.

Improve this answer
$\endgroup$
4
  • $\begingroup$ In most ANN setups,there are no bias weights (which would make them additional neurons). The biases are usually added after calculating the activation a of a neuron a = wx+b, and before applying the threshold function. $\endgroup$
    – cirko
    Commented Jul 4, 2015 at 8:55
  • $\begingroup$ My past experience was different in that people code biases as units (precisely for the reason that they can be trained like the other units). If you were to leave the bias at 1 forever you will shift the activation once caused by the initial bias weight. E.g. if the initial weight is 0.5 and you never update the bias, your threshold will always be 0.5 (think of the single layer perceptron) $\endgroup$
    – runDOSrun
    Commented Jul 4, 2015 at 9:46
  • $\begingroup$ Well now that we have two differing answers, -- and I also read a lot about ANN setups like the one you mentioned, but supposed that the one with biases for each node was the more recent one -- which one is the correct one? Or does that qualify for posting a new question :) $\endgroup$
    – cirko
    Commented Jul 5, 2015 at 10:55
  • $\begingroup$ @NeilG : for clarification of the sub-question within this question, I opened another question (by the way jubilee round lot no. 160000 ;-) ): Are fixed bias neurons or biased neurons better? $\endgroup$
    – cirko
    Commented Jul 5, 2015 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.