Why are polynomial functions bad as activations?
$\begingroup$
$\endgroup$
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
There has been some work which experiments with quadratic activations -- see "neural tensor networks" but in general a disadvantage of second order and higher polynomials is that they don't have a bounded derivative, which could lead to exploding gradients.
-
2$\begingroup$ This also applies more generally, in all kinds of maths&science domains: polynomials, especially higher ones, are dangerous unless you have an a-priori rigidly confine range of input values. Neural networks are probably one of the applications where it's least possible to give any such a-priori guarantees. $\endgroup$ Commented Jun 23, 2019 at 0:43
$\begingroup$
$\endgroup$
Nutshell So Polynomial activation functions don't work, since they fail to have the main property which makes neural networks interesting.
Mathematical Reason Actually, there is a more rigorous reason why they are not used. In this paper, it is shown that the collection of all feed-forward neural networks can approximate any (reasonable) function if and only if the activation function is not a polynomial.
Explicit Counter-Example: As an example, the simplest polynomial functions (which are non-constant) affine affine functions. If affine functions could be used (ie the universal approximation peropty were to hold) then linear regressions could approximate any continuous function. Which isn't the case.
