1
$\begingroup$

Has anyone ever tried to build a Multi-Layer Perceptron Neural Network without the sigmoid function? Let me explain better:

We know that a perceptron is a binary classifier that assign the test point to the class 0 if the dot product with the learned weight is negative, and to the class $1$ if the scalar producut is zero or positive.

Imagine now that I build a layered network, like a MLP, but without the sigmoid function for activating the neurons at each step. In this way the output of each neuron is just the dot product of the input at the previous layer.

Has this a name? What's in your opinion the best way to test this? Which framework/software/library would you use to test this?

$\endgroup$
2
$\begingroup$

Yes this does have a name: .... wait for it .... Perceptron.

Jokes aside, let me explain a little further. When you throw away the non-linearities (like sigmoid, ReLu, ...) in neural networks (as you said) the output of each neuron is just the dot product of the input at the previous layer. Thus you will only be able to express linear problems, and the neural network will no longer be a "universal function approximator".

In fact you can show that each multi-layer perceptron without non-linearities, can be reduced to a single-layer preceptron.

If you still want to implement it you can do this in any deep learning library (e.g., Keras, Tensorflow, Theano, ...).

$\endgroup$
2
$\begingroup$

If I'm not mistaken, this is just a linear combination/transformation. You just multiply every boundary by a coefficient, then you add another, without thresholds. This is entirely possible, but the "boundary" of decisions is still just a line.

By adding, multiplying and subtracting nonlinear boundaries, you can get useful, "smart" combinations. By combining lines (a*input + b) you will only ever get a line. No matter how many layers you have, a combination of lines will only ever be a line, and neural networks are based on sophisticated decision boundaries: enter image description here

By adding and subtracting lines, you will never get "bends". A useful demonstration can be found at convnetjs. Try changing the 'activation' from tanh to relu back and forth to see what the difference is. Relu produces straight corners while tanh produces blobby boundaries.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.