Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

706 questions
Filter by
Sorted by
Tagged with
61k views

### Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
100k views

3k views

### Choosing an appropriate minibatch size for stochastic gradient descent (SGD)

Is there any literature that examines the choice of minibatch size when performing stochastic gradient descent? In my experience, it seems to be an empirical choice, usually found via cross-...
1k views

### Why isn't “Saddle-Free Newton” descent algorithm used in practice?

Recently I have read a paper by Yann Dauphin et al. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, where they introduce an interesting descent ...
2k views

### Gradient descent doesn't find solution to ordinary least squares on this dataset?

I have been studying linear regression and tried it on below set {(x,y)}, where x specified the area of house in square-feet, and y specified the price in dollars. This is the first example in Andrew ...
11k views

### How does minibatch gradient descent update the weights for each example in a batch?

If we process say 10 examples in a batch, I understand we can sum the loss for each example, but how does backpropagation work in regard to updating the weights for each example? For example: ...
9k views

### Possible to evaluate GLM in Python/scikit-learn using the Poisson, Gamma, or Tweedie distributions as the family for the error distribution?

Trying to learn some Python and Sklearn, but for my work I need to run regressions that use error distributions from the Poisson, Gamma, and especially Tweedie families. I don't see anything in the ...