Suppose I have a multi-layer neural network, with non-linear activation functions, and no bias terms. Is it possible to train this network such that for any input $x$, it will always give the same (or nearly the same) output $y$?
$\begingroup$
$\endgroup$
2
asked Nov 20, 2015 at 20:45
-
$\begingroup$ What do you mean by "for any input x, it will always give the same (or nearly the same) output y"? Taking that sentence literally is incompatible to neural network functionality and pattern recognition in general. $\endgroup$– DigioCommented Nov 21, 2015 at 0:06
-
$\begingroup$ Yes, I know -- it is totally useless -- but it is just a theoretical question. Does a multi-layer neural network (without bias) have the capacity to give an output entirely independent of inputs? $\endgroup$– KarnivaurusCommented Nov 21, 2015 at 1:15
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
For tanh, relu or linear activations where all weights are 0 the output will be 0 for any input. No bias units is equivalent to bias units are all 0.
In other cases, you can use 0 weights with other activations to output specific nonzero constants; subsequent linear layers can make this constant into any fixed value. For example, logistic(0)=0.5, so you can consider linear combinations of 0.5.
$\begingroup$
$\endgroup$
Yes. This happens even when people don't want it to. It happens when people train catagorization networks with very unbalanced datasets, then network will often catagorize every input as the most common class in the training set.
It also happens in GANs, with so called "mode collapse", where the generative portion of the network learns to produce a single output that always tricks the adversarial part.
