-1
$\begingroup$

The function learned by a Deep Neural Network is essentially composition of different functions. For ex. in CNN first function is convolution (linear function), max-pooling (convex function) followed by non-linearity (sigmoid, which is convex) and so on. The basic functions are convex than how come their composition of non-convex?

I read in convex optimization that composition of convex functions is convex, they why do we always say that DNN's have non-convex energy?

$\endgroup$
  • 2
    $\begingroup$ How is a sigmoid function convex? $\endgroup$ – kjetil b halvorsen May 8 '16 at 13:45
  • 2
    $\begingroup$ @kjetil b halvorsen pointed out that the sigmoid function is not convex. Beyond that, the statement which you allegedly read "that composition of convex functions is convex" is false, unless additional assumptions (restrictions) are made. For example, exp(-x) is convex, but its composition with itself, exp(-exp(-x)) is neither convex nor concave. See section 3.2 "Operations that preserve convexity" in Boyd and Vandenberghe "Convex Optimization", which is available for free at stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf . $\endgroup$ – Mark L. Stone May 8 '16 at 15:10
  • $\begingroup$ Hi, I post an answer on similar topic here - stats.stackexchange.com/questions/106334/… I will be glad if we can chat in comments about my misunderstand (or any extra thoughts). $\endgroup$ – bruziuz Jul 10 '17 at 1:20
1
$\begingroup$

As pointed out in comments, what you have read you must have misunderstood. Apart from that, neural networks are universal approximators, that is, can approximate uniformly any continuous function on compacts (that is for one-internal layer networks, deep nets can possibly approximate even more functions). If neural networks always gave convex functions, that property would fail!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.