All Questions
Tagged with gradient optimization
12
questions
6
votes
1answer
2k views
gradient descent and local maximum
I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
5
votes
2answers
327 views
Why is two-sided gradient checking more accurate? [closed]
In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking:
One-sided difference:
$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
4
votes
1answer
2k views
Question with Matrix Derivative: Why do I have to transpose?
In the equation for Recurrent Neural Networks:
$$
h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b)
$$
Where $h_t$ is of size (N,H)
Where $W_{hh}$ is of size (H,H)
Where $W_{xh}$ is of size (D,H)
Where $...
4
votes
1answer
1k views
Expectation of gradients
I am reading this and am puzzled by equation 8. I don't understand the last bit: why can we move the gradient out of the expectation? $$E_Q[\nabla_\phi\log Q_\phi(h|x)] = E_Q[\frac{\nabla_\phi Q_\phi(...
3
votes
1answer
10k views
Gradient of loss function for (non)-linear prediction functions
$
\newcommand{\y}{\mathbf{y}}
\newcommand{\wv}{\mathbf{w}}
\newcommand{\xv}{\mathbf{x}}
\newcommand{\loss}{L(\wv;\xv, y)}
$
I'm trying to clear up the calculation of the gradient of a loss function, ...
3
votes
0answers
205 views
Gradient-informed global optimization
I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following:
A hybrid descent method ...
2
votes
3answers
978 views
Is gradient checking useless in high dimensional setting?
Is gradient checking (finite difference for numerical gradient to check if analytical gradient is correct) useless in high dimensional setting (say 100K parameters in a deep neural network)?
Here is ...
2
votes
3answers
2k views
Can I checking the correct implementation for gradient descent algorithm by looking at if the loss is monotonically decreasing?
The tricky thing of manually implement optimization algorithm is that, even there are some errors, such as wrong gradient, the algorithm still can work in some way, i.e., decrease the objective, and ...
1
vote
2answers
3k views
Calculating gradient of a function for optimization
I need to optimize a function. This function is a likelihood function which takes a set of parameters (to be optimized) and calculates the likelihood (to be optimized) as a result.
...
1
vote
0answers
200 views
How to combine gradient noise with optimization methods like Adam
I see two options:
Apply gradient noise before you apply an optimization method such as Adam (or just SGD with momentum or something else). I.e. you calculate the gradients, then you add noise to it, ...
1
vote
0answers
69 views
Gradient of a sum of indicators
EDITED w.r.t. whuber's comment: Say I have a function $\mathbb R^n \rightarrow \mathbb R$:
$$f(w_1,\ldots,w_n) = \frac{n^-\sum_{i\in I^-}w_ix_i}{n^+\sum_{i\in I^+}w_ix_i}$$
with fixed $x_i\in\mathbb ...
0
votes
1answer
32 views
Optimisation by using directional derivative
So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...
