All Questions

Tagged with
Filter by
Sorted by
Tagged with
6
votes
1answer
2k views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
5
votes
2answers
327 views

Why is two-sided gradient checking more accurate? [closed]

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking: One-sided difference: $\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
4
votes
1answer
2k views

Question with Matrix Derivative: Why do I have to transpose?

In the equation for Recurrent Neural Networks: $$ h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b) $$ Where $h_t$ is of size (N,H) Where $W_{hh}$ is of size (H,H) Where $W_{xh}$ is of size (D,H) Where $...
4
votes
1answer
1k views

Expectation of gradients

I am reading this and am puzzled by equation 8. I don't understand the last bit: why can we move the gradient out of the expectation? $$E_Q[\nabla_\phi\log Q_\phi(h|x)] = E_Q[\frac{\nabla_\phi Q_\phi(...
3
votes
1answer
10k views

Gradient of loss function for (non)-linear prediction functions

$ \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\xv}{\mathbf{x}} \newcommand{\loss}{L(\wv;\xv, y)} $ I'm trying to clear up the calculation of the gradient of a loss function, ...
3
votes
0answers
205 views

Gradient-informed global optimization

I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ...
2
votes
3answers
978 views

Is gradient checking useless in high dimensional setting?

Is gradient checking (finite difference for numerical gradient to check if analytical gradient is correct) useless in high dimensional setting (say 100K parameters in a deep neural network)? Here is ...
2
votes
3answers
2k views

Can I checking the correct implementation for gradient descent algorithm by looking at if the loss is monotonically decreasing?

The tricky thing of manually implement optimization algorithm is that, even there are some errors, such as wrong gradient, the algorithm still can work in some way, i.e., decrease the objective, and ...
1
vote
2answers
3k views

Calculating gradient of a function for optimization

I need to optimize a function. This function is a likelihood function which takes a set of parameters (to be optimized) and calculates the likelihood (to be optimized) as a result. ...
1
vote
0answers
200 views

How to combine gradient noise with optimization methods like Adam

I see two options: Apply gradient noise before you apply an optimization method such as Adam (or just SGD with momentum or something else). I.e. you calculate the gradients, then you add noise to it, ...
1
vote
0answers
69 views

Gradient of a sum of indicators

EDITED w.r.t. whuber's comment: Say I have a function $\mathbb R^n \rightarrow \mathbb R$: $$f(w_1,\ldots,w_n) = \frac{n^-\sum_{i\in I^-}w_ix_i}{n^+\sum_{i\in I^+}w_ix_i}$$ with fixed $x_i\in\mathbb ...
0
votes
1answer
32 views

Optimisation by using directional derivative

So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...