Skip to main content
Cross Validated

Return to Question

5
duplicates list edited from Neural networks: how can convex optimization produce different weights each time? to Neural networks: how can convex optimization produce different weights each time?, When will gradient descent converge to a critical point or to a local/global minima) for non-convex functions?
Link
Sycorax ♦
Sycorax ♦
  • 94.1k
  • 23
  • 236
  • 390
added 397 characters in body
Link
CCZ23
CCZ23
  • 314
  • 3
  • 12

Are convex functions pointless for Neural Networks Why are Convex Loss Functions important in SGD?

added 397 characters in body
Source Link
CCZ23
CCZ23
  • 314
  • 3
  • 12

Why are convex loss functions so important in Neural Networks? Because Neural networks are learnt end-to-end, with non-linear activations, causing convex loss functions to actually become non-convex problems.

So my issue is when using SGD optimizer, this requires a convex loss function, but if a Neural Network makes it non-convex, there will be multiple local optimas now, so why can we not just use non-convex functions instead to begin with? If it is not a convex problem, why does SGD work just for convex functions?

I feel that it is to do with the Hessian matrix. For example, am i right in saying that if there are more weights, the less likely it is that the Hessian is positive semidefinite, meaning most of the critical points of local optima will be saddle nodes which SGD can escape. Whereas, if it were non-convex, it is likely to be a positive semidefinite Hessian, meaning it will take longer to converge?

Why are convex loss functions so important in Neural Networks? Because Neural networks are learnt end-to-end, with non-linear activations, causing convex loss functions to actually become non-convex problems.

So my issue is when using SGD optimizer, this requires a convex loss function, but if a Neural Network makes it non-convex, there will be multiple local optimas now, so why can we not just use non-convex functions instead to begin with? If it is not a convex problem, why does SGD work just for convex functions?

Why are convex loss functions so important in Neural Networks? Because Neural networks are learnt end-to-end, with non-linear activations, causing convex loss functions to actually become non-convex problems.

So my issue is when using SGD optimizer, this requires a convex loss function, but if a Neural Network makes it non-convex, there will be multiple local optimas now, so why can we not just use non-convex functions instead to begin with? If it is not a convex problem, why does SGD work just for convex functions?

I feel that it is to do with the Hessian matrix. For example, am i right in saying that if there are more weights, the less likely it is that the Hessian is positive semidefinite, meaning most of the critical points of local optima will be saddle nodes which SGD can escape. Whereas, if it were non-convex, it is likely to be a positive semidefinite Hessian, meaning it will take longer to converge?

Post Closed as "Duplicate" by Sycorax♦ machine-learning
Post Reopened by Sycorax♦
added 173 characters in body
Added to review
Source Link
CCZ23
CCZ23
  • 314
  • 3
  • 12

Why are convex loss functions so important in Neural Networks? Because Neural networks are learnt end-to-end, with non-linear activations, causing convex loss functions to actually become non-convex problems. Therefore,

So my issue is when using SGD optimizer, why doesthis requires a convex loss function, but if a Neural Network makes it matter tonon-convex, there will be multiple local optimas now, so why can we not just use non-convex functions instead to begin with? If it is not a convex problem, why does SGD work just for convex functions?

Neural networks are learnt end-to-end, with non-linear activations, causing convex functions to become non-convex problems. Therefore, when using SGD, why does it matter to use convex functions?

Why are convex loss functions so important in Neural Networks? Because Neural networks are learnt end-to-end, with non-linear activations, causing convex loss functions to actually become non-convex problems.

So my issue is when using SGD optimizer, this requires a convex loss function, but if a Neural Network makes it non-convex, there will be multiple local optimas now, so why can we not just use non-convex functions instead to begin with? If it is not a convex problem, why does SGD work just for convex functions?

Post Closed as "Needs details or clarity" by Sycorax♦
Source Link
CCZ23
CCZ23
  • 314
  • 3
  • 12