I have a feed forward neural network (1 hidden layer with 10 neurons, 1 output layer with 1 neuron) with no activation function (only transfer by weight + bias) that can learn a really wonky sin wave (using a 2in1out window) with production usable accuracy trained via stochastic climbing in a couple seconds:

for (int d = 0; d < 10000; d++)
    data.Add((float)(Math.Sin((float)d * (1 / (1 + ((float)d / 300)))) + 1) / 2);

I'm probably just drunk, but if you don't use an activation function do you lose that universal function approximator status? Or is it just for gradient descent / back propagation etc. to act as a differentiable function?

Alternatively, have I probably just overlooked a bug and am actually secretly activating without knowing it?

source in C# (draws on a form)


You built a multilayer neural network with a linear hidden layer. Linear units in the hidden layer negates the purpose of having a hidden layer. The weights between your inputs and the hidden layer, and the weights between the hidden layer and the output layer are effectively a single set of weights. A neural network with a single set of weights is a linear model performing regression.

Here's a vector of your linear hidden units $$ H = [h_1, h_2,.. ,h_n] $$

The equation the governs the forward propagation of $x$ through your network is then $$ \bar{y} = W'(Hx) \Rightarrow (W'H)x $$ Thus an n-layered feed forward neural network with linear hidden layers is equivalent to a output layer given by $$ W=W'\prod_i H_i $$

If you only have linear units then the hidden layer(s) are doing nothing. Hinton et al recommends rectified linear units, which are $\text{max}(0, x)$. It's simple and doesn't suffer the vanishing gradient problem of sigmoidal functions. Similarly you might choose soft-plus function, $\log(1 + e^x)$ which is a non-sparse smooth approximation.

  • $\begingroup$ Despite of the derivation being correct, you should substitute "perceptron" with linear regression. The OP is using it to predict sine waves. A perceptron is not the right tool for this, since it is a classifier. A perceptron has the subtle different from modern neural nets in the sense that its training algorithm is not based on the formulation and differentiation of a loss function. $\endgroup$ – bayerj May 4 '14 at 20:08
  • $\begingroup$ @bayerj You're correct. Do my edits make sense? $\endgroup$ – Jessica Mick May 4 '14 at 21:56
  • $\begingroup$ Technically speaking, a perceptron is a linear model as well, i.e. linear in the parameters. You should really go with linear regression. $\endgroup$ – bayerj May 5 '14 at 6:54

If you don't have non-linear activation functions, then you end up with a network as powerful in its expressive power as a linear model. Simply view it as a linear algebra problem. Intuitively if you have linear transformation encoded by a matrix $A$ and you compose an initial vector $x$ with multiple linear transformation, then you still end up with a linear transformation:

$$ T_1( ... T_n(x) ) = A_1 \cdot ... \cdot A_n x $$

Essentially if you move points so the grids stay parallel and evenly spaced you can't randomly introduce a curve. So everything remains linear.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.