Is the cost function of a neural network with only linear activation functions a convex function with respect to its parameters? If it is, how to prove it?

  • 1
    $\begingroup$ Linear functions are closed under composition; linear regression is convex under some assumptions. $\endgroup$ – Sycorax Mar 2 at 2:11

I'm opening up the statement made in the comments.

A neural network with linear activation and many layers is equivalent to a neural network with one layer. For example:


So, in the end, your equation is $Wx+b=y$, i.e. just a linear regression. The cost function is already convex, because it is norm-squared of an affine function. Also, more strongly, the problem is a convex problem.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.