I am a new student in the world of deep learning and after studying the functioning of logistic regression and neural networks there are some insights that probably escape me.

Given these two settings:

enter image description here

I have understood how the individual steps work, from forward prop to backward prop and optimisation via gardient descent, but these are steps that are taken in both cases, so my question is:

Intuitively what is the difference? In addition to the introduction of non-linearity because of the different activation functions, is there also any change in parameters? Efficiency ? Is it more accurate?


A very basic example: logistic regression, as in your image, tries to model the class posteriors. Under no modification, the choice of nonlinearity in this case is sigmoid function; which is a linear function of inputs and neuron weights, i.e. $\sigma(w^Tx)$, where $\sigma(z)=\frac{1}{1+e^{-z}}$. Here, we set a threshold and compare the output of the activation function to decide if it is class 0 or 1; so, we decide if $\frac{1}{1+e^{-w^Tx}} > \theta$, which reduces to a inequality similar to $w^Tx < \theta_2$. This means you've just constructed a linear boundary, but nothing else. If you add a layer, the decisions you made get more complicated and not linear anymore. So, you can learn nonlinear decision boundaries.

  • $\begingroup$ So at the level of efficiency, does the neural network allow me to simply make more accurate decisions? And so these more accurate decisions derive from what? From the fact that having more layers I have more parameters ? $\endgroup$
    – Rubio95R
    Sep 13 '18 at 9:50
  • $\begingroup$ If your decision boundary is highly nonlinear, the logistic regression will perform very bad for example. Yes, there are more parameters, but the key insight is in their nonlinear interactions. One hidden layer in a NN can approximate a very good set of continuous functions, but it might need exponential number of neurons; while having two-three layer would need an order of magnitude less number of neurons to do the same job. Check out Universal Approximation Theorem also. For the downvoter, I didn't quite understand the reason of dw. Is something wrong in my explanation? $\endgroup$
    – gunes
    Sep 13 '18 at 9:59
  • 2
    $\begingroup$ This is a good answer, why was it downvoted? $\endgroup$
    – Digio
    Sep 13 '18 at 12:00
  • $\begingroup$ I have absolutely no idea. If something is wrong, it's better to at least report so that all can benefit. $\endgroup$
    – gunes
    Sep 13 '18 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.