5
$\begingroup$

While building a neural network with one hidden layer, the question arose whether or not to update the biases during backpropagation. I'm basically trying to save up on memory, so my question was and is how big a difference it would make if I updated only the weights compared to the weights and the biases. With the former, I wouldn't have to save any other bias value than the value of $1$ which I set as standard. Will it have trouble learning then? If so, then why are the weights updates insufficient for training it?

EDIT for clarity: I'm talking about the backpropagation formula

$\Delta W= -ηδ_l O_{(l-1)}$

$\Delta \theta=-ηδ_l$

Where $\Delta W$ is the difference (vector) of weights, $\Delta \theta$ is the difference (vector) of biases, $η$ is the learning rate, $O$ is the output (vector) of the layer (here $l-1$), and $\delta_l$ is the calculated error increment (vector) of layer $l$. What if you just don't use $\Delta \theta$ in backpropagation and leave the biases at $1$?

$\endgroup$
3
$\begingroup$

If you try to leave the biases fixed at any value, then each neuron will try to use its "overall input activation" as a kind of bias (by having a small weight to all of its inputs). This makes your learning less stable than just having a bias. If you try to fight this desire with regularization, it may not be possible for the network to encode a good solution to your problem.

How much memory are you really saving?

$\endgroup$
  • $\begingroup$ Well, we're talking about approx. 10^6 nodes, in that dimension. Your point seems legit. So, taken your answer and @runDOSrun 's answer together, one could say that for a stable learning function, you would either need a bias neuron with a fixed value and weights that can be updated, or biases for each neuron that can be updated as well, right? $\endgroup$ – cirko Jul 5 '15 at 10:53
  • 1
    $\begingroup$ @cirko: yes, that's the same thing. Also, a million biases is only a few megabytes of memory. $\endgroup$ – Neil G Jul 5 '15 at 15:11
4
$\begingroup$

The idea is to learn the bias weights but have the activation fixed at 1. Anything else would make it an additional ordinary unit.

$\endgroup$
  • $\begingroup$ In most ANN setups,there are no bias weights (which would make them additional neurons). The biases are usually added after calculating the activation a of a neuron a = wx+b, and before applying the threshold function. $\endgroup$ – cirko Jul 4 '15 at 8:55
  • $\begingroup$ My past experience was different in that people code biases as units (precisely for the reason that they can be trained like the other units). If you were to leave the bias at 1 forever you will shift the activation once caused by the initial bias weight. E.g. if the initial weight is 0.5 and you never update the bias, your threshold will always be 0.5 (think of the single layer perceptron) $\endgroup$ – runDOSrun Jul 4 '15 at 9:46
  • $\begingroup$ Well now that we have two differing answers, -- and I also read a lot about ANN setups like the one you mentioned, but supposed that the one with biases for each node was the more recent one -- which one is the correct one? Or does that qualify for posting a new question :) $\endgroup$ – cirko Jul 5 '15 at 10:55
  • $\begingroup$ @NeilG : for clarification of the sub-question within this question, I opened another question (by the way jubilee round lot no. 160000 ;-) ): Are fixed bias neurons or biased neurons better? $\endgroup$ – cirko Jul 5 '15 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.