Questions tagged [stochastic-gradient-descent]

Filter by
Sorted by
Tagged with
2 votes
1 answer
31 views

How to handle weighted examples in stochastic gradient descent (with mini-batches)?

Suppose I have $M$ data points $x_i$ and associated weights $w_i > 0$. I want to optimize a function, $$F(\theta) = \frac{1}{M}\sum_i w_i f(x_i;\theta)$$ in the parameters $\theta$. I will assume ...
user avatar
  • 3,348
0 votes
0 answers
7 views

True and Estimated Posterior of a Mixture of Gaussians in Bayesian Learning via SGLD

I am trying to recreate one of the experiments in this paper, (Bayesian Learning via Stochastic Gradient Langevin Dynamics). To be exact experiment 5.1. I am pretty sure, I am missing something here ...
user avatar
1 vote
0 answers
38 views

Regular *negative* spikes in neural network training loss

I am training an ALL-CNN network using the Adam solver. As the figure shows, the testing seems to converge to an acceptable solution, but there are these regular negative spikes during training that ...
user avatar
0 votes
0 answers
11 views

Mini-Batch Gradient Descent - Large batch size require small learning rate?

Coursera Machine Learning in the Enterprise - Science of Machine Learning and Custom Training says large batch size require smaller LR. However, How should the learning rate change as the batch size ...
user avatar
  • 755
1 vote
0 answers
52 views

Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network [closed]

How would I go about deriving gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward propagation expressed as: $$ F(...
user avatar
  • 11
0 votes
0 answers
14 views

why for stochastic gradient one use expectation value?

Why in order to see how far the gradient samples can be far from the true gradient, they use expectation value. For example $$E[\|\nabla f_i(w)-\nabla f(w)\|^2]=E[\|\nabla f_i(w)\|^2]-\|\nabla f(w)\|\...
user avatar
  • 133
1 vote
1 answer
23 views

How does SGD training error decrease in subsequent epochs with non-iid samples when it is recommended that samples in subsequent epochs be iid?

I have been reading the Deep Learning book by Ian Goodfellow and on pg. 277, they mention: It is also crucial that the minibatches be selected randomly. Computing an unbiased estimate of the expected ...
user avatar
0 votes
0 answers
37 views

How variable alpha changes SGDRegressor behavior for outlier?

I am using SGDRegressor with a constant learning rate and default loss function. I am curious to know how changing the alpha parameter in the function from 0.0001 to 100 will change regressor behavior....
user avatar
0 votes
0 answers
32 views

Computing gradients for Gaussian processes using locally periodic kernel

I want to use stochastic gradient descent to find hyperparameters for the locally periodic kernel. The locally periodic kernel is the product of two kernels: the periodic and squared exponential ...
user avatar
2 votes
1 answer
73 views

which training mode is more convenient for small datasets?

I have a regression problem to be solved using one of neural networks models, but I have a small dataset which contains 30 samples. Which training mode is more suitable for such dataset: stochastic or ...
user avatar
  • 153
1 vote
0 answers
28 views

Bias introduced when using weak shuffling

I have batch learning problem (in this particular case a neural network) where I am training my data in batches, and then repeating for a number of epochs. In Stochastic Gradient Descent, we minimise ...
user avatar
1 vote
0 answers
27 views

Is the difference between SGD and GD really in the summation? [duplicate]

Model: $\min _{\mathbf{w}} \frac{\lambda}{2}\|\mathbf{w}\|^{2}+\frac{1}{m} \sum_{(\mathbf{x}, y) \in S} \ell(\mathbf{w} ;(\phi(\mathbf{x}), y))$ \begin{aligned} &\text { INPUT: } S, \lambda, T \\ ...
user avatar
  • 224
0 votes
1 answer
50 views

About update procedure in data incremental learning

As far as I understood, the idea of data incremental learning consists of keeping the model always up to date. Suppose that we trained a model for user recognition using voice as input. Therefore, the ...
user avatar
  • 189
0 votes
0 answers
38 views

Effect of averaging gradients and shuffling (PPO)

In PPO (reinforcement learning algorithm) one often takes large batchsizes like 100000. To do that one averages the gradients 100 minibatches of batchsize 1000. As I understand it is recommended to ...
user avatar
2 votes
1 answer
144 views

Why does ADAM optimization perform well on non-convex functions and bad on convex functions?

I'm currently trying to understand SGD and ADAM optimization, and I understand that ADAM optimization performs well on non-convex loss functions and that SGD performs well on convex loss functions (...
user avatar
  • 217
0 votes
0 answers
48 views

Reduce Variance when using SGD in Minibatch

I am reading book: Dive into Deep Learning And I do not understand the variance in SGD for MiniBatch: The variance, on the other hand, is reduced significantly. Since the minibatch gradient is ...
user avatar
1 vote
0 answers
95 views

Does always gradients in mini-batch SGD have to be unbiased in order to prove convergence?

I am currently reading this paper [1] and [2]. The authors state that: Our analytical results include almost all of the unbiased compression techniques. And also: (i) gradient compression must be ...
user avatar
2 votes
1 answer
54 views

Stochastic Gradient Descent Code Check for Least Squares

I have an R-based implementation of the gradient descent and am trying to also get it to work as SGD. The function matches R's lm function when using the entire data set. But, when I sample from the ...
user avatar
  • 153
0 votes
1 answer
29 views

Performing Linear Regression using Stochastic Gradient Descent, by batches

I am presented with a data set, where I am supposed to perform linear regression on this using SGD. My first instinct would be to train each data point there is until I reach the last one. Only then ...
user avatar
  • 7,575
4 votes
1 answer
165 views

Why is a 2nd order derivative optimization better for no hidden layer neural networks?

I was reading in this blog. That first order derivative SGD optimization methods are worse for neural networks without hidden layers and 2nd order is better, because that's what regression uses. Why ...
user avatar
0 votes
1 answer
83 views

Why does stochastic gradient descent lead us to a minimum at all?

Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
user avatar
  • 481
2 votes
1 answer
190 views

The reason (and intuition) behind why stochastic gradient descent can get stuck on a local minimum

Suppose you want to find $k$ that minimises your cost function $J(k)$. We may want to apply batch gradient descent or stochastic gradient descent. Let's deliberately initialise $k$ with the same ...
user avatar
  • 481
0 votes
0 answers
16 views

Can stochastic gradient descent for Bayesian Inference? [duplicate]

I was looking at the Bayesian MAP estimate formula which is the "argmax(likelihood * prior)". Can this be calculated using stochastic gradient descent? Gradient descent requires knowing the ...
user avatar
  • 5,728
0 votes
1 answer
37 views

Is outer product of marginal distribution the "best" mean-field approximation for a joint distribution?

I am certain this has been asked somewhere else, if that's the case, link me and close the thread. I am studying variational inference and mean-field approximation. All the explanations I come across ...
user avatar
  • 103
2 votes
1 answer
27 views

Why a minimiser of a subset of training dataset is that of the whole training set

In section 3.3 of Bottou et al (2018), under the 'intuitive motivation' paragraph, the authors claim that 'a minimiser of empirical risk for the larger set $S$ is clearly given by a minimiser for the ...
user avatar
  • 308
2 votes
1 answer
252 views

upsampling vs class weights in mini-batch SGD

Let's consider using mini-batch SGD in (neural network) binary classification problem with imbalanced dataset. Let's say that the ratio between the number of examples in each class is positive:...
user avatar
  • 41
1 vote
0 answers
370 views

When are very small learning rates useful?

I just wondered if there are cases where small or very small learning rates in gradient descent based optimization are useful? A large learning rate allows the model to explore a much larger portion ...
user avatar
  • 615
0 votes
0 answers
37 views

Gradient Descent Algorithm for Interdependent parameters

Suppose I have $n$ data points ($X_i$,$y_i$) where $X_i$ is a vector and $y_i$ is a scalar, $1 \le i \le n$. By defining $\hat{\boldsymbol{Y}} = \boldsymbol{\Theta} \boldsymbol{X} + \boldsymbol{b}$ ...
user avatar
0 votes
0 answers
23 views

How does AdaDelta "learn" the learning rate?

In the context of training a neural net using stochastic gradient descent, you optimise parameters $ \theta $ by iteratively feeding data through the net, calculating the gradient $ \nabla P$ of the ...
user avatar
  • 215
0 votes
1 answer
180 views

Loss values above 1.0

I have a convolutional neural network for tensors classification in Pytorch. I am using Cross-Entropy Loss. My optimizer is Stochastic Gradient Descent and the learning rate is 0.0001. The accuracy of ...
user avatar
  • 31
0 votes
0 answers
53 views

Can a variable batch-size be problematic?

I am currently facing an issue with one of the models I am working on (it's a Transducer model but that does not really matter for the question in general). The problem is that the model sometimes ...
user avatar
2 votes
0 answers
82 views

Why adversarial training prefer SGD over Adam?

Why adversarial training methods (e.g. Trades) use SGD as optimizer rather than Adam?
user avatar
  • 21
2 votes
1 answer
248 views

Relationship between variance of gradient and SGD convergence

I've found things such as the Robbins-Monroe conditions for the learning rate, as well as a proof from Robbins, Siegmund, 1971 which gives convergence to a local minima provided that the expectation ...
user avatar
  • 163
1 vote
1 answer
88 views

Train a model on batches with multiple epochs vs. each batch with multiple epochs

In traditional supervised learning, say we split the data into $B$ batches, we will go through all these batches with $E$ epochs. The computational cost is $E\times B$. But what if on each batch, we ...
user avatar
2 votes
0 answers
82 views

What is the difference between implicit and explicit stochastic gradient descent?

While reading the implicit stochastic gradient descent, I get stuck by its update: $$w^{new}:=w^{old} - \eta \nabla Q_{i} (w^{new}) $$ I learned from this slide that the above is the implicit style of ...
user avatar
  • 5,181
0 votes
1 answer
413 views

In what way are SGD and Perceptron very similar?

I'm reading Hands-On Machine Learning and the author states that: You may have noticed the fact that the Perceptron learning algorithm strongly resembles Stochastic Gradient Descent. In fact, Scikit-...
user avatar
0 votes
0 answers
30 views

Least squares fit with polynomials: Order of the polynomial vs accuracy of the predictions

I have noisy evaluations $y_i$ of some unkown function $f$ at points $\vec{x}_i$ clustered around point $\vec{x^*}$. Now I want to fit a polynomial model to this data to get some surrogate model $\hat{...
user avatar
0 votes
1 answer
436 views

Implementing Stochastic Gradient Descent with both Weight Decay and Momentum

So I'm trying to implement a neural network using only numpy module in Python. The problem I'm facing is related to the correct implementation of the regularization through weight decay, and also the ...
user avatar
1 vote
0 answers
28 views

Stochastic Gradient Descent - Least Squares [closed]

I am not sure if I implemented the SGD in a proper way since in calculations it gives way to big error even on the training set. Can you help me to figure out where I made a mistake? Here $D$ is the ...
user avatar
  • 157
1 vote
1 answer
320 views

SGD unbiased estimator: 1 example vs larger minibatch for each iteration

Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. ...
user avatar
0 votes
0 answers
93 views

When is a high learning rate for Stochastic Gradient Descent a good thing?

I was always under the impression that SGD needed a lower learning rate than optimizers like Adam, because it was stochastic and more likely to make training diverge with higher learning rates. I ...
user avatar
  • 619
2 votes
1 answer
291 views

SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
user avatar
  • 43
3 votes
2 answers
2k views

Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
user avatar
  • 141
6 votes
1 answer
522 views

SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
user avatar
  • 1,124
1 vote
1 answer
205 views

Optimization of Linear Autoencoder with SGD

I'm interested in the Linear Autoencoder(LAE), and I knew that, at convergence point, the subspace LAE learns is the same as the subspace PCA learns up to linear transformations. Also, the loss ...
user avatar
  • 127
3 votes
1 answer
162 views

What is the probability distribution of a minibatch of data?

Suppose there are numbers $\{1, \ldots, 10\}$ You pick one at random, call it $i$ Then $i$ is a Uniform random variable (https://en.wikipedia.org/wiki/Discrete_uniform_distribution), $i \sim U\{1, 10\}...
user avatar
  • 674
2 votes
1 answer
34 views

can we perform sub-gradient updates in mini-batches

We are already aware that in case the data is quite bulky, mini-batch gradient descent based approaches may be applied. These approaches load a mini-batch of data, compute the loss on this batch, and ...
user avatar
  • 1,586
0 votes
1 answer
19 views

Anybody know of good material or videos to help me understand Stochastic Gradient Descent?

I am trying to understand stochastic gradient descent a bit better as I'm not 100. Does anybody have any materials or videos that they would recommend to me that might help describe the concept? I'm ...
3 votes
1 answer
216 views

Problems, which are difficult for SGD

I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also ...
user avatar
  • 31
3 votes
0 answers
98 views

How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
user avatar
  • 571