129 views

### Can we apply analyticity of a neural network to improve upon gradient descent? [duplicate]

Gradient descent uses the first order derivative information of the objective function as a function of the parameters. Gradient descent therefore uses only “local” information about the objective ...
98 views

### Second derivative test for machine learning algorithms [duplicate]

I have a question on second derivative test for most "modern" machine learning algorithms. I learned that in calculus but never seen it in real applications. Most machine learning algorithms ...
39 views

### Gradient Descent vs Newton's Method? [duplicate]

Can someone please elaborate on Gradient Descent vs Newton's Method? I am studying for my ML exam tomorrow and saw this as a bullet point but cannot find many great answers. Can anyone fill me in some ...
173k views

### What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
10k views

### Why use gradient descent with neural networks?

When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to ...
1k views

### Loss function in machine learning - how to constrain?

My loss has two parts, say L1 and L2. I want to minimize both, and at the same time I need to constrain that L1 should be always greater than L2 (L1>L2). Is the following correct? loss = L2 - L1
4k views

### Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
2k views

### Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
333 views

### Looking for book recommendations for numerical optimization

I was reading the answers and comments to this question: Why is Newton's method not widely used in machine learning? and realised that I would like to learn a lot more about numerical optimization....
1k views

### Why do saddle points become “attractive” in Newtonian dynamics?

I am reading Identifying and attacking the saddle point problem in high-dimensional non-convex optimization by Dauphin et. al. and the first paragraph on the second page states the following: A ...
651 views

I am using ADAM as an optimization algorithm to minimize some black box function $f(x,y)$. I know this function is convex and has a minimum $f(5,5) = 0$. Initially, the algorithm proceeds as expected:...