# Are there any “convex neural networks”?

Are there any neural network training procedure that involves solving a convex problem?

Note that I am referring more to MLPs, instead of (multi-class) logistic regression which is a neural network with no hidden layers.

I know that for MLPs, if there are no activation function in between (e.g. an identity activation function), then the entire model is simply $$\hat y = W_n \cdots W_1 x$$, $$x$$ is your example, $$\hat y$$ is your output, and obviously leads to a convex problem (linear regression) $$\min_w \|\hat y - y\|$$.

$$\hat y = \text{softmax}(W_n \cdots W_1 x)$$ is also convex, I believe (a composition of convex functions).

• What about the case when there are nonlinearity in between (or at the output)? Does adding ANY standard choices of nonlinearity automatically lead to a non-convex problem?

• In the same vein, are there any convex models except for (multi-class) logistic regression and MLPs with no hidden layers?