I'm curious to know how important the bias node is for the effectiveness of modern neural networks. I can easily understand that it can be important in a shallow network with only a few input variables. However, modern neural nets such as in deep learning often have a large number of input variables to decide whether a certain neuron is triggered. Would simply removing them from, e.g., LeNet5 or ImageNet have any real impact at all?
$\begingroup$
$\endgroup$
5
-
$\begingroup$ @gung - I've seen you've edited the title to use the phrase "bias node". I am curious as to why you prefer that title? I've never heard that usage before. Moreover, it seems confusing to use the word "node" when the bias is not a separate node in a network. $\endgroup$– pirCommented Dec 27, 2015 at 0:43
-
2$\begingroup$ If you dislike it, you can roll back the edit w/ my apologies. I always thought the name was fairly standard, although I haven't played w/ ANNs in years & some call it the "bias neuron" instead. FWIW, "bias" is a bit ambiguous in statistics / ML; it most commonly refers to an estimator whose sampling distribution is not centered on the true value of the parameter, or a predictive function / predicted value that differs from the true function / mean, etc., whereas the bias node is a specific part of an ANN. $\endgroup$– gung - Reinstate MonicaCommented Dec 27, 2015 at 0:52
-
2$\begingroup$ It is an actual node--at least in the sense that any of them are--in the network. Eg, see the black nodes in this image. $\endgroup$– gung - Reinstate MonicaCommented Dec 27, 2015 at 0:53
-
$\begingroup$ Okay, that makes sense - it is true that "bias" is quite ambiguous. Thanks for the explanation. $\endgroup$– pirCommented Dec 28, 2015 at 15:54
-
1$\begingroup$ For neurons the bias unit seems expontaneous firing, this happens in nature. $\endgroup$– user3927612Commented Jan 28, 2016 at 23:54
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
8
Removing the bias will definitely affect the performance and here is why...
Each neuron is like a simple logistic regression and you have $y=\sigma(W x + b)$. The input values are multiplied with the weights and the bias affects the initial level of squashing in the sigmoid function (tanh etc.), which results the desired the non-linearity.
For example, assume that you want a neuron to fire $y\approx1$ when all the input pixels are black $x\approx0$. If there is no bias no matter what weights $W$ you have, given the equation $y=\sigma(W x)$ the neuron will always fire $y\approx0.5$.
Therefore, by removing the bias terms you would substantially decrease your neural network's performance.
answered May 25, 2015 at 20:00
-
2$\begingroup$ Thanks, that makes some sense. I guess that even though most modern nets use ReLU as the activation function (see e.g. papers.nips.cc/paper/4824-imagenet), this could still be relevant if the net needed to fire when all the input pixels are black. ReLU is defined as f(x) = max(0, x). $\endgroup$– pirCommented May 25, 2015 at 20:41
-
$\begingroup$ exactly! it's the same case... $\endgroup$ Commented May 25, 2015 at 21:09
-
4$\begingroup$ The plot depicts the activations of a sigmoid neuron given the input. Now assume that we want a neuron to fire $y\approx1$ when all pixels are black $x\approx0$. This would be impossible without the bias term and when $x\approx0$ it will always fire $y\approx0.5$. $\endgroup$ Commented Jun 3, 2015 at 14:39
-
2$\begingroup$ While I agree with the theory, it is worth pointing out that with modern large nets the chances of getting an all-zero input is negligible. This also relies on the assumption that a net would want to fire a 1 - deep nets will most likely not care about the single neuron output - this is partly why dropout is so popular for regularizing nets. $\endgroup$ Commented Dec 26, 2015 at 23:34
-
2$\begingroup$ @MaxGordon is right. This answer doesn't apply to this question. Try removing the bias from a big network and you'll see that it makes very little difference. $\endgroup$– Neil GCommented Jan 29, 2016 at 1:46
$\begingroup$
$\endgroup$
2
I disagree with the other answer in the particular context of your question. Yes, a bias node matters in a small network. However, in a large model, removing the bias inputs makes very little difference because each node can make a bias node out of the average activation of all of its inputs, which by the law of large numbers will be roughly normal. At the first layer, the ability for this to happens depends on your input distribution. For MNIST for example, the input's average activation is roughly constant.
On a small network, of course you need a bias input, but on a large network, removing it makes almost no difference. (But, why would you remove it?)
-
$\begingroup$ I don't know if I get it right but what comes to my mind is the following: For example, say a neuron receives an input $x_1, \ldots x_n$ and computes $w_1 x_1 \ldots + \ldots w_n x_n$ (omitting bias). How this results to a constant term (bias)? $\endgroup$– ado sarCommented May 28, 2023 at 22:38
-
$\begingroup$ @adosar I'm suggesting that as n goes to infinity, there exists some set of weights that produces a roughly constant term. It's not hard to test this supposition. $\endgroup$– Neil GCommented May 29, 2023 at 17:24
$\begingroup$
$\endgroup$
1
I'd comment on @NeilG's answer if I had enough reputation, but alas...
I disagree with you, Neil, on this. You say:
... the average activation of all of its inputs, which by the law of large numbers will be roughly normal.
I'd argue against that, and say that the law of large number necessitates that all observations are independent of each other. This is very much not the case in something like neural nets. Even if each activation is normally distributed, if you observe one input value as being exceptionally high, it changes the probability of all the other inputs. Thus, the "observations", in this case, inputs, are not independent, and the law of large numbers does not apply.
Unless I'm not understanding your answer.
-
$\begingroup$ You're right that the naive average of all inputs may not be a good proxy for a bias node. However, all any neuron needs to do is fine some projection of inputs whose output is roughly constant to simulate a bias. Every input is another degree of freedom, and for ordinary data I think there's bound to be some projection that works. $\endgroup$– Neil GCommented May 29, 2023 at 4:35