Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [adagrad]

AdaGrad (for adaptive gradient algorithm) is an enhanced stochastic gradient descent algorithm that automatically determines a per-parameter learning rate.

Filter by
Sorted by
Tagged with
0
votes
0answers
32 views

Accelerated Projected SGD under box constraints

Are there generalizations of ADAM or Adagrad algorithm that allow box constraints for the parameters to be incorporated in the gradient descent step? Is it valid to simply run the algorithm as usual ...
0
votes
1answer
92 views

Same exact model: converges with adagrad, diverges with adadelta

I have a very simple model built using Keras. What strikes me as surprising is that the very same training configs converge (i.e. training loss goes down with every epoch) when the model uses the <...
1
vote
0answers
107 views

Behavior of AdaGrad without the square root in the denominator

Multiple articles claim that AdaGrad does not work well when the square-root in the formula is not taken. This is one such example. $\theta_{t+1,i} = \theta_{t,i}-\dfrac{\eta}{\sqrt{G_{t,ii}+\epsilon}...
1
vote
1answer
176 views

Intuition behind learning rate scheduling in AdaDelta

To get rid of the problems in AdaGrad, the learning rate is changed from $\frac{\eta}{\sqrt{G_{t, ii}+\epsilon}}$ to include only gradients in a small window size $w$. But as the function approaches ...
1
vote
0answers
182 views

Why does Adagrad improve the robustness of SGD?

I mainly read this blog. And this blog sites this paper for the statement that Adagrad improved the robustness of SGD. I have tried to check the original paper or other articles which explains why ...
1
vote
1answer
111 views

Adagrad Expression about Element-wise matrix vector multiplication

Sometimes, Adagrad is expressed like this $\mathbf{x}^{t+1} = \mathbf{x}^t –[{η/√{G^t + ε}}]$ ⊙ ∇E where G is a diagonal matrix. Accoding to wiki, Hadamard product is only defined when two matrxes ...
4
votes
1answer
473 views

Should you use optimization algorithms like Adagrad and ADAM for neural network online training?

Optimization algorithms like Adagrad and ADAM decay your learning rate over time. To me this sounds like a bad idea for online training since you're always getting new data as opposed to retraining on ...
0
votes
1answer
699 views

xgboost: get error for each iteration

in XGBoost, is there a way to programmatically get the training and evaluation error per iteration of training? ...
1
vote
0answers
153 views

Treating Categorical Variables as Continuous for Random Forest / Adaboost

What's the correct way to deal with categorical variables in packages like sklearn's RF and xgboost? Is there any cons of treating the variables are continuous? E.g. encode class A as 1, class B as ...
1
vote
0answers
41 views

What happens if we use training data in reverse chronological order?

My SGD-Adagrad algorithm uses chronological data for training for future predictions. The test and validation data has occured after training data. What happens if I use training data in reverse ...
0
votes
1answer
636 views

Divergence in Stochastic Gradient Descent

I am using Stochastic Gradient Descent with ADAGRAD. I am training on a training set of 1.6 billion examples. After about 30 million examples, the training loss starts increasing after reaching a low....
2
votes
1answer
691 views

How to perform adagrad stochastic gradient descent (SGD) on word2vec?

AdaGrad is an enhanced SGD that automatically determines a per-parameter learning rate. However, in word2vec, there's no clear "parameter" to perform adagrad on. So what's the closest algorithm to ...
5
votes
0answers
446 views

Adagrad for batch gradient descent

There are many papers on how Adagrad is used in SGD, but I have not seen any where it is applied in batch descent. I have a situation wherein batch gradient descent is faster than SGD (unique to my ...