# All Questions

Tagged with
12 questions
Filter by
Sorted by
Tagged with
2k views

In the equation for Recurrent Neural Networks: $$h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b)$$ Where $h_t$ is of size (N,H) Where $W_{hh}$ is of size (H,H) Where $W_{xh}$ is of size (D,H) Where $... 1answer 32 views ### Optimisation by using directional derivative So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ... 0answers 205 views ### Gradient-informed global optimization I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ... 1answer 2k views ### gradient descent and local maximum I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ... 2answers 350 views ### Why is two-sided gradient checking more accurate? [closed] In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking: One-sided difference:$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - ...
202 views

I see two options: Apply gradient noise before you apply an optimization method such as Adam (or just SGD with momentum or something else). I.e. you calculate the gradients, then you add noise to it, ...
979 views

### Is gradient checking useless in high dimensional setting?

Is gradient checking (finite difference for numerical gradient to check if analytical gradient is correct) useless in high dimensional setting (say 100K parameters in a deep neural network)? Here is ...