Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

704 questions
Filter by
Sorted by
Tagged with
2k views

Finding the Peak of a Kernel Density Estimator

I implemented a Kernel Density Estimator. I have a multivariate dataset that I use with it and I would like to find the point of highest likelihood. A way I thought about is sampling n points using ...
42k views

Why use gradient descent for linear regression, when a closed-form math solution is available?

I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis. h(x) = B0 + B1X why we ...
15 views

1k views

Are Residual Networks related to Gradient Boosting?

Recently, we saw the emergence of the Residual Neural Net, wherein, each layer consists of a computational module $c_i$ and a shortcut connection that preserves the input to the layer such as the ...
46 views

using finite difference to estimate high dimensional gradient in gradient descent methods

I'm not very familiar with optimization problem, but I know that if the gradient of function is hard to find, it can use finite difference method to estimate it. Like scipy.minimize, it would use this ...
39 views

Why using gradient decent if we can just minimize function in closed form?

I don't really understand why gradient descent is so important in neural networks? Wouldn't it be much easier to define an objective function in a way to do ...
524 views

Mini-batching dependent data

I have a neural network I am training on some time series data. Naturally I want to sequentially mini-batch this data if at all possible. However, it seems that if the data size isn't a multiple of ...
2k views

How to make stochastic gradient descent algorithm converge to the optimum?

(Background info taken from my blog) In logistic regression, the hypothesis function, which models the relationshiop between the dependent variable $P(y = 1)$ and the independent variable $X$, is : ...
652 views

Motivation behind parameter sharding for Downpour SGD

Why does the Downpour model shard the parameters into separate groups? Is there any advantage of making one cluster responsible for changing only certain parameters?
624 views

what is the correct formula of momentum for gradient descent?

I have been trying to get a better understanding of momentum, but in my search for clarification I got pretty confused. The main reason is that there seem to be multiple different, non-equivalent ...
8 views

Gradient of a convex loss for linear classifiers

Let L(w; x; y) be a convex loss function for a linear classifier w. Can you always express the gradient of ...
473 views

RMSProp and Adam both scale the effectively learning rate by dividing the moving average of past gradients (root mean squared). So if the first layer has gradient much smaller than the last layer, ...
29 views

Interpreting cost change plot in a neural network for learning XOR

I tried to build a neural net for learning XOR. The design is as follows: 1st layer: compute linear function of input 4:2 with 2:2 weights and adding 1:2 bias. 2nd layer: apply sigmoid to all ...
98 views

Can we apply analyticity of a neural network to improve upon gradient descent? [duplicate]

Gradient descent uses the first order derivative information of the objective function as a function of the parameters. Gradient descent therefore uses only “local” information about the objective ...
31 views

Why is the step length by default equal to 1 in gradient boosting?

On ESL p.359, it explains steepest descent: But in 10.37, it is trying to minimize the distance to g_im. It looks like the default step length is 1. Why is it so?
22 views

Logistic regression fitting methods clarification

Each book I read propose a different fitting method for Logistic Regression. The general idea is to maximize this expression.  Pr\left(\beta|y,X,M\right) = \frac{Pr\left(y|\beta,X,M\right) Pr(\...
29 views

Decision tree that fits a regression at leaf nodes

Is there any academic work on any Decision Tree that fits a regression at its final leaf node? For instance, suppose I have 100 features (X), and use them build a tree with 3 depths such that I have ...
186 views

In Goodfellow et al.'s Deep Learning, the authors write, "rectified linear units use the activation function $g(z) = \max\{0,z\}$... The gradients are not only large but consistent" (187). What does ...
31 views

Neural network training: skipping layers when performing weight updates

Suppose you had many models that are trained to classify a sequence of logical characters into 'satisfiable' or 'unsatisfiable' (e.g., the sequence "A and notA" is unsatisfiable). Suppose all of these ...
72 views

Why is local-m Attention differentiable?

In this paper (https://arxiv.org/pdf/1508.04025.pdf) local attention is introduced. With local-m attention, we compute the attention vector over the source hidden states within a window around the ...
309 views

Proximal Gradient Descent and Proximal Coordinate descent for Lasso Problem

Why is proximal coordinate descent much less affected by bad conditioning than proximal gradient descent? For example, we can consider this problem : $\min_x \frac{1}{2}\|Ax-b\|^2_2 + \lambda\|x\|_1$ ...
It has been decades since I coded up any type of gradient descent algorithm to drive a function to zero (or to a minimum). I am following this tutorial, which minimizes $J(\overrightarrow{\theta})$. ...