15 questions
Filter by
Sorted by
Tagged with
35 views

1 vote
39 views

### Momentum vs adaptive step methods

My understanding is that: With momentum, one can avoid e.g. "zig-zags" during gradient decent by averaging gradients to determine a better direction of descent. With adaptive step size ...
936 views

### Momentum vs Polyak averaging

I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers. For example, is it correct to say that ...
1 vote
137 views

I have a very simple model built using Keras. What strikes me as surprising is that the very same training configs converge (i.e. training loss goes down with every epoch) when the model uses the <...
1 vote
218 views

Multiple articles claim that AdaGrad does not work well when the square-root in the formula is not taken. This is one such example. $\theta_{t+1,i} = \theta_{t,i}-\dfrac{\eta}{\sqrt{G_{t,ii}+\epsilon}... 1 vote 1 answer 294 views ### Intuition behind learning rate scheduling in AdaDelta To get rid of the problems in AdaGrad, the learning rate is changed from$\frac{\eta}{\sqrt{G_{t, ii}+\epsilon}}$to include only gradients in a small window size$w$. But as the function approaches ... 1 vote 0 answers 208 views ### Why does Adagrad improve the robustness of SGD? I mainly read this blog. And this blog sites this paper for the statement that Adagrad improved the robustness of SGD. I have tried to check the original paper or other articles which explains why ... 1 vote 1 answer 165 views ### Adagrad Expression about Element-wise matrix vector multiplication Sometimes, Adagrad is expressed like this$\mathbf{x}^{t+1} = \mathbf{x}^t –[{η/√{G^t + ε}}]\$ ⊙ ∇E where G is a diagonal matrix. Accoding to wiki, Hadamard product is only defined when two matrxes ...
487 views

Optimization algorithms like Adagrad and ADAM decay your learning rate over time. To me this sounds like a bad idea for online training since you're always getting new data as opposed to retraining on ...
887 views

### xgboost: get error for each iteration

in XGBoost, is there a way to programmatically get the training and evaluation error per iteration of training? ...
1 vote
339 views

### Treating Categorical Variables as Continuous for Random Forest / Adaboost

What's the correct way to deal with categorical variables in packages like sklearn's RF and xgboost? Is there any cons of treating the variables are continuous? E.g. encode class A as 1, class B as ...
1 vote
43 views

### What happens if we use training data in reverse chronological order?

My SGD-Adagrad algorithm uses chronological data for training for future predictions. The test and validation data has occured after training data. What happens if I use training data in reverse ...
1k views

### Divergence in Stochastic Gradient Descent

I am using Stochastic Gradient Descent with ADAGRAD. I am training on a training set of 1.6 billion examples. After about 30 million examples, the training loss starts increasing after reaching a low....