Importance of the bias node in neural networks I'm curious to know how important the bias node is for the effectiveness of modern neural networks. I can easily understand that it can be important in a shallow network with only a few input variables. However, modern neural nets such as in deep learning often have a large number of input variables to decide whether a certain neuron is triggered. Would simply removing them from, e.g., LeNet5 or ImageNet have any real impact at all?
 A: I'd comment on @NeilG's answer if I had enough reputation, but alas... 
I disagree with you, Neil, on this. You say:

... the average activation of all of its inputs, which by the law of large numbers will be roughly normal.

I'd argue against that, and say that the law of large number necessitates that all observations are independent of each other. This is very much not the case in something like neural nets. Even if each activation is normally distributed, if you observe one input value as being exceptionally high, it changes the probability of all the other inputs. Thus, the "observations", in this case, inputs, are not independent, and the law of large numbers does not apply.
Unless I'm not understanding your answer. 
A: Removing the bias will definitely affect the performance and here is why... 
Each neuron is like a simple logistic regression and you have $y=\sigma(W x + b)$. The input values are multiplied with the weights and the bias affects the initial level of squashing in the sigmoid function (tanh etc.), which results the desired the non-linearity.
For example, assume that you want a neuron to fire $y\approx1$ when all the input pixels are black $x\approx0$. If there is no bias no matter what weights $W$ you have, given the equation $y=\sigma(W x)$ the neuron will always fire $y\approx0.5$.

Therefore, by removing the bias terms you would substantially decrease your neural network's performance.
A: I disagree with the other answer in the particular context of your question.  Yes, a bias node matters in a small network.  However, in a large model, removing the bias inputs makes very little difference because each node can make a bias node out of the average activation of all of its inputs, which by the law of large numbers will be roughly normal.  At the first layer, the ability for this to happens depends on your input distribution.  For MNIST for example, the input's average activation is roughly constant.
On a small network, of course you need a bias input, but on a large network, removing it makes almost no difference.  (But, why would you remove it?)
