Is the cost function of a neural network with only linear activation functions a convex function with respect to its parameters? If it is, how to prove it?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ Linear functions are closed under composition; linear regression is convex under some assumptions. $\endgroup$– Sycorax ♦Commented Mar 2, 2020 at 2:11
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I'm opening up the statement made in the comments.
A neural network with linear activation and many layers is equivalent to a neural network with one layer. For example:
$$W_2(W_1x+b_1)+b_2=Wx+b$$
So, in the end, your equation is $Wx+b=y$, i.e. just a linear regression. The cost function is already convex, because it is norm-squared of an affine function. Also, more strongly, the problem is a convex problem.
