All Questions
Tagged with gradient optimization
14
questions
7
votes
1answer
3k views
gradient descent and local maximum
I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
5
votes
2answers
579 views
Why is two-sided gradient checking more accurate? [closed]
In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking:
One-sided difference:
$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
5
votes
1answer
2k views
Question with Matrix Derivative: Why do I have to transpose?
In the equation for Recurrent Neural Networks:
$$
h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b)
$$
Where $h_t$ is of size (N,H)
Where $W_{hh}$ is of size (H,H)
Where $W_{xh}$ is of size (D,H)
Where $...
5
votes
1answer
1k views
Expectation of gradients
I am reading this and am puzzled by equation 8. I don't understand the last bit: why can we move the gradient out of the expectation? $$E_Q[\nabla_\phi\log Q_\phi(h|x)] = E_Q[\frac{\nabla_\phi Q_\phi(...
5
votes
1answer
12k views
Gradient of loss function for (non)-linear prediction functions
$
\newcommand{\y}{\mathbf{y}}
\newcommand{\wv}{\mathbf{w}}
\newcommand{\xv}{\mathbf{x}}
\newcommand{\loss}{L(\wv;\xv, y)}
$
I'm trying to clear up the calculation of the gradient of a loss function, ...
4
votes
0answers
270 views
Gradient-informed global optimization
I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following:
A hybrid descent method ...
2
votes
3answers
3k views
Can I checking the correct implementation for gradient descent algorithm by looking at if the loss is monotonically decreasing?
The tricky thing of manually implement optimization algorithm is that, even there are some errors, such as wrong gradient, the algorithm still can work in some way, i.e., decrease the objective, and ...
2
votes
3answers
1k views
Is gradient checking useless in high dimensional setting?
Is gradient checking (finite difference for numerical gradient to check if analytical gradient is correct) useless in high dimensional setting (say 100K parameters in a deep neural network)?
Here is ...
2
votes
0answers
233 views
How to combine gradient noise with optimization methods like Adam
I see two options:
Apply gradient noise before you apply an optimization method such as Adam (or just SGD with momentum or something else). I.e. you calculate the gradients, then you add noise to it, ...
1
vote
2answers
3k views
Calculating gradient of a function for optimization
I need to optimize a function. This function is a likelihood function which takes a set of parameters (to be optimized) and calculates the likelihood (to be optimized) as a result.
...
1
vote
0answers
71 views
Gradient of a sum of indicators
EDITED w.r.t. whuber's comment: Say I have a function $\mathbb R^n \rightarrow \mathbb R$:
$$f(w_1,\ldots,w_n) = \frac{n^-\sum_{i\in I^-}w_ix_i}{n^+\sum_{i\in I^+}w_ix_i}$$
with fixed $x_i\in\mathbb ...
0
votes
1answer
34 views
Optimisation by using directional derivative
So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...
0
votes
0answers
4 views
can alternating optimization be performed in mini-batches
Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version).
Although I am perfectly fine with the full batch version ...
0
votes
0answers
100 views
Valid use of differentiable almost everywhere functions, like hinge loss, in gradient optimization/learners, like SciKit-Learn's SGDClassifier?
So, my abstract question is: is it valid (in the sense that stable convergence is roughly expected) to use functions that are differentiable almost everywhere in the practical application of ...
