Two-part question: Neural Networks(NN) can be looked at as stacked units of logistic regression classifiers (LRC). A basic requirement of an activation function is to be non-linear. When LRC is a neuron, sigmoid function is the activation function and is said to bring non-linearity to NN.

Q1:) If this is true, why is LRC still a linear classifier even when it uses non-linear sigmoid function?

Further, if the activation function is a linear function(e.g. an identity function), the NN can no longer learn non-linear decision boundaries.

Q2:) Does this mean that the depth of a neural network plays no role in making it non-linear?

I have scanned similar threads for this question. But not convinced yet!

  • 1
    $\begingroup$ What's the definition of a linear function? How can you apply that definition to the particular case of the neural networks you're considering? $\endgroup$
    – Sycorax
    Dec 8, 2020 at 17:42
  • $\begingroup$ Hi, thanks, I edited my answer a bit. A linear function's graph would have a straight line, it's output can be expressed as a sum of it's inputs (could be scaled). I say if we use the identity function as an activation function, the final output for an input 'x' can be expressed as Wx + B, where W encompasses matrix products of all Ws of all layers of NN. Same goes for B. In my opinion, that does make an NN linear when a linear activation function is used. $\endgroup$ Dec 8, 2020 at 17:58
  • $\begingroup$ Yes, the composition of several linear operations is itself linear. Some more detail can be found here stats.stackexchange.com/questions/228296/… $\endgroup$
    – Sycorax
    Dec 8, 2020 at 18:02
  • $\begingroup$ Thanks. Is it okay to conclude that the non-linearity of a NN arises due to the combination of its depth and non-linear activation functions? Hence, either of the two alone will not suffice. $\endgroup$ Dec 8, 2020 at 18:16
  • 1
    $\begingroup$ @Sycorax Ah! I see. Such overloadings have been a constant source of paradoxical intuitions for me. Thanks a lot for clarifying that they exist outside my mind too. $\endgroup$ Dec 8, 2020 at 18:27