Suppose I have a multi-layer neural network, with non-linear activation functions, and no bias terms. Is it possible to train this network such that for any input $x$, it will always give the same (or nearly the same) output $y$?

  • $\begingroup$ What do you mean by "for any input x, it will always give the same (or nearly the same) output y"? Taking that sentence literally is incompatible to neural network functionality and pattern recognition in general. $\endgroup$
    – Digio
    Nov 21, 2015 at 0:06
  • $\begingroup$ Yes, I know -- it is totally useless -- but it is just a theoretical question. Does a multi-layer neural network (without bias) have the capacity to give an output entirely independent of inputs? $\endgroup$ Nov 21, 2015 at 1:15

2 Answers 2


For tanh, relu or linear activations where all weights are 0 the output will be 0 for any input. No bias units is equivalent to bias units are all 0.

In other cases, you can use 0 weights with other activations to output specific nonzero constants; subsequent linear layers can make this constant into any fixed value. For example, logistic(0)=0.5, so you can consider linear combinations of 0.5.


Yes. This happens even when people don't want it to. It happens when people train catagorization networks with very unbalanced datasets, then network will often catagorize every input as the most common class in the training set.

It also happens in GANs, with so called "mode collapse", where the generative portion of the network learns to produce a single output that always tricks the adversarial part.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.