Questions tagged [adagrad]

AdaGrad (for adaptive gradient algorithm) is an enhanced stochastic gradient descent algorithm that automatically determines a per-parameter learning rate.

Filter by
Sorted by
Tagged with
0 votes
1 answer
35 views

When do Adaptive Optimization Algorithms modify their parameters?

When do "Ada" optimizers (e.g. Adagrad, Adam, etc...) "adapt" their parameters? Is it at the end of each mini-batch or epoch?
user avatar
1 vote
0 answers
39 views

Momentum vs adaptive step methods

My understanding is that: With momentum, one can avoid e.g. "zig-zags" during gradient decent by averaging gradients to determine a better direction of descent. With adaptive step size ...
user avatar
  • 3,528
2 votes
0 answers
936 views

Momentum vs Polyak averaging

I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers. For example, is it correct to say that ...
user avatar
  • 3,528
1 vote
1 answer
137 views

Same exact model: converges with adagrad, diverges with adadelta

I have a very simple model built using Keras. What strikes me as surprising is that the very same training configs converge (i.e. training loss goes down with every epoch) when the model uses the <...
user avatar
  • 1,000
1 vote
0 answers
218 views

Behavior of AdaGrad without the square root in the denominator

Multiple articles claim that AdaGrad does not work well when the square-root in the formula is not taken. This is one such example. $\theta_{t+1,i} = \theta_{t,i}-\dfrac{\eta}{\sqrt{G_{t,ii}+\epsilon}...
user avatar
1 vote
1 answer
294 views

Intuition behind learning rate scheduling in AdaDelta

To get rid of the problems in AdaGrad, the learning rate is changed from $\frac{\eta}{\sqrt{G_{t, ii}+\epsilon}}$ to include only gradients in a small window size $w$. But as the function approaches ...
user avatar
1 vote
0 answers
208 views

Why does Adagrad improve the robustness of SGD?

I mainly read this blog. And this blog sites this paper for the statement that Adagrad improved the robustness of SGD. I have tried to check the original paper or other articles which explains why ...
user avatar
1 vote
1 answer
165 views

Adagrad Expression about Element-wise matrix vector multiplication

Sometimes, Adagrad is expressed like this $\mathbf{x}^{t+1} = \mathbf{x}^t –[{η/√{G^t + ε}}]$ ⊙ ∇E where G is a diagonal matrix. Accoding to wiki, Hadamard product is only defined when two matrxes ...
user avatar
4 votes
1 answer
487 views

Should you use optimization algorithms like Adagrad and ADAM for neural network online training?

Optimization algorithms like Adagrad and ADAM decay your learning rate over time. To me this sounds like a bad idea for online training since you're always getting new data as opposed to retraining on ...
user avatar
0 votes
1 answer
887 views

xgboost: get error for each iteration

in XGBoost, is there a way to programmatically get the training and evaluation error per iteration of training? ...
user avatar
1 vote
0 answers
339 views

Treating Categorical Variables as Continuous for Random Forest / Adaboost

What's the correct way to deal with categorical variables in packages like sklearn's RF and xgboost? Is there any cons of treating the variables are continuous? E.g. encode class A as 1, class B as ...
user avatar
1 vote
0 answers
43 views

What happens if we use training data in reverse chronological order?

My SGD-Adagrad algorithm uses chronological data for training for future predictions. The test and validation data has occured after training data. What happens if I use training data in reverse ...
user avatar
  • 145
0 votes
1 answer
1k views

Divergence in Stochastic Gradient Descent

I am using Stochastic Gradient Descent with ADAGRAD. I am training on a training set of 1.6 billion examples. After about 30 million examples, the training loss starts increasing after reaching a low....
user avatar
  • 145
2 votes
1 answer
862 views

How to perform adagrad stochastic gradient descent (SGD) on word2vec?

AdaGrad is an enhanced SGD that automatically determines a per-parameter learning rate. However, in word2vec, there's no clear "parameter" to perform adagrad on. So what's the closest algorithm to ...
user avatar
5 votes
0 answers
475 views

Adagrad for batch gradient descent

There are many papers on how Adagrad is used in SGD, but I have not seen any where it is applied in batch descent. I have a situation wherein batch gradient descent is faster than SGD (unique to my ...
user avatar
  • 2,224