While studying as a newbie about Neural Networks I started as everyone from the basics (perceptrons, MLPs) then how backpropagation works before dive in to harder deep learning concepts.

Now, I am trying to solve exercices (theoretical and practical) in order to practice and I have two questions.

1) I read that the Perceptron (single neuron) relates to a logistic regression classifier in particular, a (binary) logistic regression classifier is the same as a (single neuron) Perceptron with a sigmoid activation function. Is it true? How can I prove it? any link/example with examples etc.

2) A Multi-layer NN without activation function is equivalent to applying a linear transformation to the input data. Can someome explain me why this is true? an example?


1 Answer 1


1- We need to show that the 2 scalar softmax outputs of binary logistic regression are not necessary, and that you can make the exact same predictions using a single sigmoid output.

In a binary logistic regression classifier, you have two scalar outputs: $S(x) = [S(x)_0,\hspace{0.2cm}S(x)_1]$, where $S$ is the softmax function. The prediction would then be the class0 if $S(x)_0 > S(x)_1$, and class1 if $S(x)_0 < S(x)_1$. Since we know that $S(x)_0 + S(x)_1 = 1$ (property of softmax), then we can say that the prediction would be class0 if $S(x)_0 > 0.5$ and class1 if $S(x)_0 < 0.5$ (you can check that this is equivalent to the previous statement). Already you can see that we do not actually need $S(X)_1$ to make a prediction, but we can make a prediction having only the value of $S(X)_0$. Now let's see how we can calculate the value of $S(X)_0$ using sigmoid: $$ S(x)_0 = \dfrac{e^{x_0}}{e^{x_0}+e^{x_1}}= \dfrac{1}{1+\frac{e^{x_1}}{e^{x_0}}}= \dfrac{1}{1+e^{-(x_0-x_1)}}=\sigma(z)\quad\text{(where $z=x_0-x_1$)} $$ $$z=x_0-x_1=(w_0-w_1)a+(b_0-b_1)=w'a+b'$$

So all you need to do is use a single node with the weight vector $w'=w_0-w_1$ and the bias $b'=b_0-b_1$ to calculate $z$ (where $w_0$ and $w_1$ are the weights corresponding to the two nodes of the binary logistic regression model), and then proceed to take the sigmoid of $z$.

2- Suppose a 3 layer neural network without activation functions: $$ A_1 = W_1X\\ A_2 = W_2A_1\\ A_3 = W_3A_2 $$ $X$ is our input matrix where each column contains an input sample, $W_i$ is the weight matrix, and $A_i$ is the output at the $i$'th layer. Then we can simply write $A_3$ as: $$ A_3 = W_{3}(A_2) = W_{3}(W_{2}A_1)= W_{3}(W_{2}(W_1X))=(W_{3}W_{2}W_1)X= W'X $$ The output ($W'X$) is just a linear transformation ($W'$) applied to your inputs ($X$).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.