# Linked Questions

1answer
129 views

### Can we apply analyticity of a neural network to improve upon gradient descent? [duplicate]

Gradient descent uses the first order derivative information of the objective function as a function of the parameters. Gradient descent therefore uses only “local” information about the objective ...
0answers
98 views

### Second derivative test for machine learning algorithms [duplicate]

I have a question on second derivative test for most "modern" machine learning algorithms. I learned that in calculus but never seen it in real applications. Most machine learning algorithms ...
0answers
39 views

### Gradient Descent vs Newton's Method? [duplicate]

Can someone please elaborate on Gradient Descent vs Newton's Method? I am studying for my ML exam tomorrow and saw this as a bullet point but cannot find many great answers. Can anyone fill me in some ...
8answers
173k views

### What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
3answers
10k views

### Why use gradient descent with neural networks?

When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to ...
4answers
1k views

### Loss function in machine learning - how to constrain?

My loss has two parts, say L1 and L2. I want to minimize both, and at the same time I need to constrain that L1 should be always greater than L2 (L1>L2). Is the following correct? loss = L2 - L1
2answers
4k views

### Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
2answers
2k views

### Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
2answers
333 views

### Looking for book recommendations for numerical optimization

I was reading the answers and comments to this question: Why is Newton's method not widely used in machine learning? and realised that I would like to learn a lot more about numerical optimization....
1answer
1k views

### Why do saddle points become “attractive” in Newtonian dynamics?

I am reading Identifying and attacking the saddle point problem in high-dimensional non-convex optimization by Dauphin et. al. and the first paragraph on the second page states the following: A ...
1answer
651 views

### ADAM Gradient descent oscillates close to minimum

I am using ADAM as an optimization algorithm to minimize some black box function $f(x,y)$. I know this function is convex and has a minimum $f(5,5) = 0$. Initially, the algorithm proceeds as expected:...
1answer
372 views

### Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1answer
220 views

### Are there are alternatives to gradient update rule?

Most optimization techniques (that I'm aware of) for non-linear cost functions that are commonly implemented rely on linearly updating a variable iteratively until a minimum is reached or a condition ...