Questions tagged [sgd]

Stochastic gradient descent (SGD) is a variant of gradient descent where only a small subset ("mini-batch") of training examples is used to compute the gradient on each iteration.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

High initial test validation score for Neural Network

From only the first epoch of training my NN with SGD (I use Xavier initialisation for weights), the accuracy shoots off to 92%, and then flattens out. The same thing happens with loss (but lower, of ...
0
votes
0answers
37 views
+50

Choosing learning rate with 2nd order method - minimizing parabola in one step?

In parabola $(\theta,g)$ values are in line $(g=f'(\theta))$ - we can get slope of this line e.g. by dividing their standard deviations: $$ \mu = \frac{\sigma_\theta}{\sigma_g}=\sqrt{\frac{var(\...
1
vote
0answers
8 views

Modified Loss Function(s) for decorrelating neurons within a layer?

I'm looking for previous references on a specific topic. Does anyone know of any modified loss functions that incentivize a network to produce a diagonal neuron-to-neuron covariance matrix (averaging ...
0
votes
0answers
11 views

How to diagnose convergence in CF for low-rank matrix factorization trained with SGD?

I have a huge dataset of user-item ratings and would like to make predictions. To this end I'm using the low-rank matrix factorization with SGD for CF (collaborative filtering). I pre-compute the ...
0
votes
0answers
15 views

Learning Curve for SGD [closed]

I am dealing with a binary classification problem where and I am trying a suite of ML models and testing their performance on my data. Data: consisting of 804 rows of different columns. Target ...
2
votes
1answer
24 views

Why is it hard to bring back a once “broken” neural network to usable results ? - Classification MNIST SGD

I'm currently working through Michael Nielsens book "Neural Networks and Deep Learning" . If I use his code/hyperparameters the network get's quite a good classification score. Even after the first ...
0
votes
0answers
11 views

Stochastic/batch gradient descent (type SGD/ADAM) with weighted mean square error loss

Assume I assign uneven weights to losses of different examples, i.e. I set my SGD/ADAM to train a universal approximator f (e.g. a neural net) by minimization of a weighted mean square error: $ L = \...
0
votes
0answers
17 views

For large $n$, what tuning parameters need to be changed in SGD?

In stochastic descent algorithms, what happens if we increase the data size that we are working with? Does this cause a change to the tuning parameters that we can use? For example, say you are ...
0
votes
2answers
57 views

Why do people say gradient descent is slower than stochastic gradient descent? That's obviously not true?

With gradient descent, you calculate the gradient for the entire sample at once. With SGD, you calculate it on each sample, and then you do the same for every other sample, until you have done 1 full ...
0
votes
2answers
33 views

XOR with Neural Network [closed]

I'm trying to implement a simple neural network to fit a XOR function as shown in the book 'Deep Learning' by Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016). Here is my python code using ...
2
votes
1answer
38 views

Stopping criteria for stochastic gradient descent?

When using stochastic gradient descent, how do we pick a stopping criteria? A benefit of stochastic gradient descent is that, since it is stochastic, it can avoid getting stuck in a suboptimal ...
0
votes
0answers
73 views

Bayesian interpretation of gradient clipping

In the context of Bayesian interpretations of SGD for neural network training, is there an interpretation for the gradient clipping operation which is often included?
0
votes
0answers
83 views

Time complexity of batch gradient descent

I am read http://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf paper. It states that "Due to the poor condition number, the standard batch ...
2
votes
1answer
36 views

The correct implementation of momentum method and NAG

Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of ...
3
votes
0answers
47 views

Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
0
votes
0answers
15 views

Why LMS filter can be regarded as SGD?

My question is about LMS algorithm. Why say LMS filter is Stochastic Gradient Descent? Can I apply SVRG (Stochastic Variance reduced gradient) for LMS filter? And...What's wrong with this code? <...
1
vote
0answers
17 views

Why picking several times of same instances generally converge faster than going through instance by instance using Stochastic Gradient Descent?

I am reading Hands-on Machine Learning with Scikit-Learn & TensorFlow by Aurelien Geron. In chapter 4: Training models page 122, where it is explaining linear regression using SGD, it says that ...
2
votes
4answers
153 views

How does stochastic gradient descent even work for neural nets?

How does stochastic gradient descent (meaning where you backpropagate and adjust the weights and biases of the neural network after each single sample) even work? Doesn't that just tell the neural ...
2
votes
1answer
83 views

Can you use stochastic gradient descent with a multinomial likelihood?

I have a multinomial likelihood of the form: $$P(\underline n|\underline x) = N!\prod_{i=1}^M \frac{f_i(\underline x)^{n_i}}{n_i!}$$ where $\underline x$ is a vector of parameters, $f_i(\underline x)...
0
votes
0answers
32 views

What's the bug in my implementation/understanding of backpropagation?

For learning purposes, I'm trying to implement a simple neural network with only linear layers followed by logistic activation. As far as I understand, the backpropagation algorithm exploits the ...
0
votes
0answers
98 views

Variance when using mini-batch stochastic gradient descent

When using mini-batch stochastic gradient descent, we have the option of either summing the observation specific gradient estimates, or we can take an average over the observation-specific gradient ...
0
votes
0answers
14 views

SGD versus Adamax on XOR operator

I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ...
1
vote
1answer
73 views

What does it mean when the global gradient norm keeps decreasing while loss has converged?

I am training an autoencoder with a $L^2$ loss. Training gives reasonable results and the loss quickly converge to a non-zero but acceptable value after about 5 epochs: However, looking at the "...
0
votes
0answers
47 views

Is Stochastic Gradient Descent sensitive to training permutation?

I've recently read that SGD (Stochastic Gradient Descent) is one of the most popular techniques for training Machine Learning algorithms, including DNNs (deep neural networks). However, my ...
0
votes
0answers
43 views

What is the relationship between stochastic mirror descent and stochastic gradient descent?

I don't know much about stochastic mirror descent and was wondering if someone could briefly summarize it in general terms and compare/contrast it to stochastic gradient descent. When I understand ...
3
votes
1answer
86 views

Difference between eligibility traces and momentum?

Eligibility traces and function approximators. I'm looking at Sutton & Barto's use of eligibility traces combined with function approximation (e.g. sections 13.5, 13.6) and I noticed that it ...
1
vote
1answer
523 views

Combining Random forest with Adam (or an other gradient method)

There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD? Can I use Adam (Adaptive moment estimation)...
0
votes
0answers
39 views

Minimizing expected loss with non-fixed probability distribution

Is there any convergence studies or algorithm to solve the following problem? $$ \mathbf{\hat{w}} = \min_\mathbf{w} \int\mathcal{L}(\mathbf{x};\mathbf{w})P(\mathbf{x};\mathbf{w})\ \mathrm{d}\mathbf{x}...
2
votes
1answer
364 views

Does Keras SGD optimizer implement batch, mini-batch, or stochastic gradient descent?

I am a newbie in Deep Learning libraries and thus decided to go with Keras. While implementing a NN model, I saw the batch_size parameter in ...
0
votes
0answers
37 views

Stochastic Gradient Descent: using minibatch AND small dataset size

My dataset has textual samples where the number of the samples is <1000. I'm training an NERecogniser by using spaCy library. Is using minibatch important in my case (small dataset size)? I ...
6
votes
1answer
328 views

Can small SGD batch size lead to faster overfitting?

I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...
0
votes
0answers
24 views

Singular values for a latent-factor model

Suppose we build a latent-factor model using alternating least squares (ALS) or stochastic gradient descent (SGD). Can we calculate weights for each latent factor, in a similar way to how the singular-...
1
vote
0answers
36 views

Momentum updates average of g, Adagrad also of g^2 - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
1
vote
0answers
81 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
0answers
28 views

What are good packages for online linear regression besides Vowpal Wabbit?

Does anyone know of online learning packages that implement NG and NAG algorithms from Stephen Ross' paper: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://auai.org/uai2013/prints/papers/...
0
votes
0answers
86 views

Multiclass hinge loss gradient

I am trying to compute the gradient of multi class hinge loss function but i am kinda confused. First things first, I have a W matrix [10xD] (10 classes) that contains the weights. The loss ...
0
votes
0answers
17 views

How to get precise answer from stochastic gradient descent

I have a convex optimization problem in few variables and I have an unbiased estimator of the gradient without having the ability to evaluate the function itself. I want to do gradient descent but the ...
4
votes
1answer
323 views

Is stochastic gradient descent pseudo-stochastic?

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times. ...
1
vote
0answers
83 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
5
votes
1answer
163 views

Do there exist adaptive step size methods for Newton-Raphson optimization?

Stochastic/Mini-batch gradient descent, caused by interest in deep learning, has made lots of advances in adaptive step sizes. For example, Adam, Nadam, Adamax, ..., are all improvements to the ...
8
votes
1answer
1k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
0
votes
1answer
57 views

Is training loss guaranteed to decrease for stochastic gradient descent? [duplicate]

When performing stochastic gradient descent, it is necessary for the training loss to decrease a) between iterations in an epoch? (I think the answer is no) b) between epochs? (I think the answer is ...
5
votes
0answers
23 views

Why divide the learning rate by the size of the mini batch? [duplicate]

In Michael Nielsen's online book Neural Networks and Deep Learning, in chapter one (and onwards) he divides the learning rate, $\eta$, by the size of the mini batch when he performs stochastic ...
1
vote
0answers
36 views

Tensorflow InvalidArgumentError: The determinant is not finite [closed]

I'm trying to fit a Mixture of Gaussians to a data set. First the data is clustered using K-Means Clustering. Each cluster is then fitted with a Gaussian.To avoid inversion of large covariance matrix, ...
0
votes
0answers
34 views

Accelerated Projected SGD under box constraints

Are there generalizations of ADAM or Adagrad algorithm that allow box constraints for the parameters to be incorporated in the gradient descent step? Is it valid to simply run the algorithm as usual ...
2
votes
1answer
1k views

When does my autoencoder start to overfit?

I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has ...
0
votes
0answers
117 views

Natural Gradients in Stochastic Variational Inference (SVI) for Gaussian Process Regression

Currently, I've hard times in understanding the natural gradients update in SVI method for Gaussian Process. I'm learning the SVI method for Gaussian Process through Gaussian Process for Big Data ...
3
votes
1answer
587 views

Why increasing the batch size has the same effect as decaying the learning rate?

There have been a few papers this year, concerned with very large scale training, where instead than decaying the learning rate $\eta$, the batch size $B$ was increased, usually with the same schedule ...
1
vote
1answer
71 views

How to understand whether Stochastic Gradient Descent has converged?

I am using SGD to solve for MSE function. My training set is around 50K, and I am monitoring the gradient at every epoch (once a pass is completed over all the training data). I played around a lot ...
0
votes
0answers
156 views

How to choose the learning rate for stochastic gradient descent (via backtracking)?

I am trying to implement "from scratch" SGD and Mini Batch Gradient Descent in Matlab. I have to minimize a function like $f(x)= \sum f_i (X_i, y_i)$ where $(X_i, y_i)$ is a data point (features and ...