Questions tagged [stochastic-gradient-descent]

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

SGD with momentum update - CS231n explanation

In the notes for Stanford's CS231n course there is an explanation for the Momentum update. I'm confused by the usage of the word "integrates" here, e.g. "gradient directly integrates ...
7
votes
5answers
1k views

For regression with time varying parameters, SGD or Kalman filter?

What is the advantage of kalman filters as an online update mechanism instead of the stochastic gradient descent?
134
votes
4answers
131k views

Batch gradient descent versus stochastic gradient descent

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
0
votes
0answers
5 views

Epoch Metrics vs. Step Metrics?

I'm running PyTorch and I'm trying to log metrics. I just have one question - When using mini-batch gradient descent, and logging metrics in the training loop, you can get the training and validation ...
1
vote
0answers
22 views

Stochastic Gradient Descent - Least Squares [closed]

I am not sure if I implemented the SGD in a proper way since in calculations it gives way to big error even on the training set. Can you help me to figure out where I made a mistake? Here $D$ is the ...
0
votes
0answers
30 views

Implementing Stochastic Gradient Descent with both Weight Decay and Momentum

So I'm trying to implement a neural network using only numpy module in Python. The problem I'm facing is related to the correct implementation of the regularization through weight decay, and also the ...
2
votes
0answers
376 views

How to get SGD to reach global optimal point in logistic regression?

I am trying to write a tool which involves implementing logistic regression. With the batch gradient descent method, the convergence is guaranteed as it is a convex problem. However, I find that with ...
1
vote
1answer
44 views

SGD unbiased estimator: 1 example vs larger minibatch for each iteration

Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. ...
0
votes
1answer
276 views

How to deal with numeric instability in stochastic gradient descent?

Imagine that we try to perform sgd using a gradient that takes very small or very large values (e.g. it is a product of many terms that are larger than 1). Is there a standard approach to deal with ...
0
votes
0answers
18 views

When is a high learning rate for Stochastic Gradient Descent a good thing?

I was always under the impression that SGD needed a lower learning rate than optimizers like Adam, because it was stochastic and more likely to make training diverge with higher learning rates. I ...
25
votes
2answers
13k views
0
votes
0answers
16 views

Reinforcement Learning: SGD sampling and the independence of samples in sequences

I am taking a course in RL and many times, learning policy parameters of value function weights essentially boils down to using Stochastic Gradient Descent (SGD). The agent is represented as having a ...
3
votes
1answer
139 views

constant terms in stochastic gradient descent: when to apply them and how much of the constant gradient component?

in a derivation for the gradient of a collaborative filtering system (similar to Probabilistic Matrix Factorization), I got to the following expression for the gradient of a latent vector $\mathbf{u}...
0
votes
0answers
10 views

Why do we need more than 1 epoch to train data? [duplicate]

As 1 epoch means each data point has gone through the algorithm once and made changes in weighted values accordingly . So , why there is a need to process same data again and again ? How does it ...
1
vote
1answer
129 views

Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
2
votes
1answer
59 views

SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
5
votes
1answer
253 views

SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
0
votes
0answers
21 views

Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...
2
votes
3answers
791 views

Is a loss function computed after each step of gradient descent or after a whole epoch?

In neural networks with mini-batch or stochastic gradient descent, is a loss function computed after each step of gradient descent or after a whole epoch?
1
vote
1answer
56 views

Optimization of Linear Autoencoder with SGD

I'm interested in the Linear Autoencoder(LAE), and I knew that, at convergence point, the subspace LAE learns is the same as the subspace PCA learns up to linear transformations. Also, the loss ...
6
votes
1answer
1k views

Mean or sum of gradients for weight updates in SGD

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
0
votes
0answers
15 views

can alternating optimization be performed in mini-batches

Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version). Although I am perfectly fine with the full batch version ...
3
votes
1answer
78 views

What is the probability distribution of a minibatch of data?

Suppose there are numbers $\{1, \ldots, 10\}$ You pick one at random, call it $i$ Then $i$ is a Uniform random variable (https://en.wikipedia.org/wiki/Discrete_uniform_distribution), $i \sim U\{1, 10\}...
2
votes
1answer
16 views

can we perform sub-gradient updates in mini-batches

We are already aware that in case the data is quite bulky, mini-batch gradient descent based approaches may be applied. These approaches load a mini-batch of data, compute the loss on this batch, and ...
6
votes
1answer
171 views

Difference between eligibility traces and momentum?

Eligibility traces and function approximators. I'm looking at Sutton & Barto's use of eligibility traces combined with function approximation (e.g. sections 13.5, 13.6) and I noticed that it ...
0
votes
1answer
17 views

Anybody know of good material or videos to help me understand Stochastic Gradient Descent?

I am trying to understand stochastic gradient descent a bit better as I'm not 100. Does anybody have any materials or videos that they would recommend to me that might help describe the concept? I'm ...
5
votes
1answer
196 views

Equivalent Gradients in Kernelized SVM

Let $\varphi: \mathcal{X} \to \mathcal{H}$ a mapping with corresponding kernel $K:\mathcal{X}\times\mathcal{X}\to \mathbb{R}$ (that is, $K\left(x,x'\right) = \left<\varphi\left(x\right), \varphi\...
3
votes
1answer
146 views

Problems, which are difficult for SGD

I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also ...
3
votes
0answers
44 views

How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
19
votes
2answers
4k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
1
vote
2answers
1k views

The actual role of second-order optimization as oppose to first-order optimizations

I do not fully understand how second-order optimization approaches help machine learning algorithms, like multilayer perceptron, to achieve the global minimum error....
0
votes
0answers
61 views

Why not use line search in conjunction with stochastic gradient descent?

I'm familiar with numerical optimization in Engineering context. I have taken several graduate level engineering optimization and operations research courses. I'm beginning to learn machine learning. ...
1
vote
0answers
74 views

Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
2
votes
1answer
366 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
1answer
31 views

When can I say a network converges better?

I trained two networks with and without skip connection. The network is a FCNN and has an encoding-decoding structure. I trained the networks with SGD and MSE for 150 epochs. The attached image is a ...
1
vote
1answer
40 views

How can I improve a classification algorithm for dogs and cats?

The following code is a ML algorithm trained to classify between dogs and cats, the database is composed by 25000 images (evenly split) and can be obtained at this Link (if you click it will ...
0
votes
0answers
7 views

Difference between using SGD on new data in batches vs training on the whole dataset (recommender systems)

I work with recommender systems and use SGD to train them. I am doing both: real-time updates: updating weights as soon as a new batch of 64 entries come in and training on the whole (well part of ...
2
votes
1answer
150 views

Matrix factorization for expanding matrix

In the paper Matrix Factorization Techniques for Recommender Systems Koren, Bell and Volinsky describe how the matrix $R_{n \times k}$ (users $\times$ movie ratings) can be decomposed to $P_{n \times ...
1
vote
1answer
26 views

Convergence under large set of learning rates

What is the interpretation of a stochastic optimization problem where a gradient descent algorithm is converging under a wide range of learning rate schedules (including ones with quite large initial ...
1
vote
0answers
37 views

Batch size influence on model quality

I found in https://dl.acm.org/doi/abs/10.1145/3320060 (section 3) this graph that illustrates influence of batch size. Below there is an explanation: We can show the existence of region C by ...
1
vote
1answer
2k views

stochastic gradient descent of ridge regression when regularization parameter is very big

As we know, the gradient of ridge regression is: $$ g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta $$ where $X_i$ is the $i$th training sample. The update of $\theta$ is ...
26
votes
6answers
7k views

For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
1
vote
1answer
279 views

What is neural network good accuracy

I am very new at machine learning, and I'm building an artificial neural network that aims to classify inputs into 2 labels. I am training the network with randomly initialized weights and through the ...
1
vote
0answers
86 views

On projected gradient descent and inequality constraints

Consider the optimization problem \begin{equation} \min_{x\in\mathbb{R}^n} \quad f(x) \end{equation} using the gradient descent, we can iteratively solve this problem \begin{equation} x^{k+1} = x^k-\...
3
votes
2answers
764 views

Projected Gradient Descent

Consider the primal SVM problem: $$ \frac12||w||^2 +\frac Cm \sum_{i=1}^m \max(0,1-y_iw\cdot x_i) $$ We want to find a solution with a bounded norm, by using SGD with a projection onto the convex set:...
0
votes
1answer
178 views

How to define nearest neighbor search such that it can be optimized using stochastic gradient descent?

Assume that there is a reference two-dimensional array ref and a given vector x. I would like to return the closest vector to <...
26
votes
4answers
34k views

How does batch size affect convergence of SGD and why?

I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
2
votes
1answer
1k views

Difference between stochastic variational inference and variational inference?

Very simple, as the question header says: what is the difference between SVI and VI? I cannot seem to find a clear-cut definition online.
1
vote
0answers
21 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...