5
$\begingroup$

I have a feed forward neural network (1 hidden layer with 10 neurons, 1 output layer with 1 neuron) with no activation function (only transfer by weight + bias) that can learn a really wonky sin wave (using a 2in1out window) with production usable accuracy trained via stochastic climbing in a couple seconds:

for (int d = 0; d < 10000; d++)
    data.Add((float)(Math.Sin((float)d * (1 / (1 + ((float)d / 300)))) + 1) / 2);

I'm probably just drunk, but if you don't use an activation function do you lose that universal function approximator status? Or is it just for gradient descent / back propagation etc. to act as a differentiable function?

Alternatively, have I probably just overlooked a bug and am actually secretly activating without knowing it?

source in C# (draws on a form)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
10
$\begingroup$

You built a multilayer neural network with a linear hidden layer. Linear units in the hidden layer negates the purpose of having a hidden layer. The weights between your inputs and the hidden layer, and the weights between the hidden layer and the output layer are effectively a single set of weights. A neural network with a single set of weights is a linear model performing regression.

Here's a vector of your linear hidden units $$ H = [h_1, h_2,.. ,h_n] $$

The equation the governs the forward propagation of $x$ through your network is then $$ \bar{y} = W'(Hx) \Rightarrow (W'H)x $$ Thus an n-layered feed forward neural network with linear hidden layers is equivalent to a output layer given by $$ W=W'\prod_i H_i $$

If you only have linear units then the hidden layer(s) are doing nothing. Hinton et al recommends rectified linear units, which are $\text{max}(0, x)$. It's simple and doesn't suffer the vanishing gradient problem of sigmoidal functions. Similarly you might choose soft-plus function, $\log(1 + e^x)$ which is a non-sparse smooth approximation.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Despite of the derivation being correct, you should substitute "perceptron" with linear regression. The OP is using it to predict sine waves. A perceptron is not the right tool for this, since it is a classifier. A perceptron has the subtle different from modern neural nets in the sense that its training algorithm is not based on the formulation and differentiation of a loss function. $\endgroup$
    – bayerj
    May 4, 2014 at 20:08
  • $\begingroup$ @bayerj You're correct. Do my edits make sense? $\endgroup$
    – Jessica Collins
    May 4, 2014 at 21:56
  • $\begingroup$ Technically speaking, a perceptron is a linear model as well, i.e. linear in the parameters. You should really go with linear regression. $\endgroup$
    – bayerj
    May 5, 2014 at 6:54
  • $\begingroup$ You mean soft-max not soft-plus? $\endgroup$
    – kjetil b halvorsen ♦
    Oct 5, 2020 at 0:10
4
$\begingroup$

If you don't have non-linear activation functions, then you end up with a network as powerful in its expressive power as a linear model. Simply view it as a linear algebra problem. Intuitively if you have linear transformation encoded by a matrix $A$ and you compose an initial vector $x$ with multiple linear transformation, then you still end up with a linear transformation:

$$ T_1( ... T_n(x) ) = A_1 \cdot ... \cdot A_n x $$

Essentially if you move points so the grids stay parallel and evenly spaced you can't randomly introduce a curve. So everything remains linear.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.