# Does a Neural Network actually need an activation function or is that just for Back Propagation?

I have a feed forward neural network (1 hidden layer with 10 neurons, 1 output layer with 1 neuron) with no activation function (only transfer by weight + bias) that can learn a really wonky sin wave (using a 2in1out window) with production usable accuracy trained via stochastic climbing in a couple seconds:

for (int d = 0; d < 10000; d++)
data.Add((float)(Math.Sin((float)d * (1 / (1 + ((float)d / 300)))) + 1) / 2);


I'm probably just drunk, but if you don't use an activation function do you lose that universal function approximator status? Or is it just for gradient descent / back propagation etc. to act as a differentiable function?

Alternatively, have I probably just overlooked a bug and am actually secretly activating without knowing it?

source in C# (draws on a form)

You built a multilayer neural network with a linear hidden layer. Linear units in the hidden layer negates the purpose of having a hidden layer. The weights between your inputs and the hidden layer, and the weights between the hidden layer and the output layer are effectively a single set of weights. A neural network with a single set of weights is a linear model performing regression.

Here's a vector of your linear hidden units $$H = [h_1, h_2,.. ,h_n]$$

The equation the governs the forward propagation of $x$ through your network is then $$\bar{y} = W'(Hx) \Rightarrow (W'H)x$$ Thus an n-layered feed forward neural network with linear hidden layers is equivalent to a output layer given by $$W=W'\prod_i H_i$$

If you only have linear units then the hidden layer(s) are doing nothing. Hinton et al recommends rectified linear units, which are $\text{max}(0, x)$. It's simple and doesn't suffer the vanishing gradient problem of sigmoidal functions. Similarly you might choose soft-plus function, $\log(1 + e^x)$ which is a non-sparse smooth approximation.

• Despite of the derivation being correct, you should substitute "perceptron" with linear regression. The OP is using it to predict sine waves. A perceptron is not the right tool for this, since it is a classifier. A perceptron has the subtle different from modern neural nets in the sense that its training algorithm is not based on the formulation and differentiation of a loss function. May 4, 2014 at 20:08
• @bayerj You're correct. Do my edits make sense? May 4, 2014 at 21:56
• Technically speaking, a perceptron is a linear model as well, i.e. linear in the parameters. You should really go with linear regression. May 5, 2014 at 6:54
• You mean soft-max not soft-plus? Oct 5, 2020 at 0:10

If you don't have non-linear activation functions, then you end up with a network as powerful in its expressive power as a linear model. Simply view it as a linear algebra problem. Intuitively if you have linear transformation encoded by a matrix $A$ and you compose an initial vector $x$ with multiple linear transformation, then you still end up with a linear transformation:

$$T_1( ... T_n(x) ) = A_1 \cdot ... \cdot A_n x$$

Essentially if you move points so the grids stay parallel and evenly spaced you can't randomly introduce a curve. So everything remains linear.