While building a neural network with one hidden layer, the question arose whether or not to update the biases during backpropagation. I'm basically trying to save up on memory, so my question was and is how big a difference it would make if I updated only the weights compared to the weights and the biases. With the former, I wouldn't have to save any other bias value than the value of 1 which I set as standard. Will it have trouble learning then? If so, then why are the weights updates insufficient for training it?

While building a neural network with one hidden layer, the question arose whether or not to update the biases during backpropagation. I'm basically trying to save up on memory, so my question was and is how big a difference it would make if I updated only the weights compared to the weights and the biases. With the former, I wouldn't have to save any other bias value than the value of $1$ which I set as standard. Will it have trouble learning then? If so, then why are the weights updates insufficient for training it?

EDIT for clarity: I'm talking about the backpropagation formula

$\Delta W= -ηδ_l O_{(l-1)}$

$\Delta \theta=-ηδ_l$

Where $\Delta W$ is the difference (vector) of weights, $\Delta \theta$ is the difference (vector) of biases, $η$ is the learning rate, $O$ is the output (vector) of the layer (here $l-1$), and $\delta_l$ is the calculated error increment (vector) of layer $l$. What if you just don't use $\Delta \theta$ in backpropagation and leave the biases at $1$?