# Questions tagged [stochastic-gradient-descent]

The tag has no usage guidance.

159 questions
Filter by
Sorted by
Tagged with
0answers
14 views

### SGD with momentum update - CS231n explanation

In the notes for Stanford's CS231n course there is an explanation for the Momentum update. I'm confused by the usage of the word "integrates" here, e.g. "gradient directly integrates ...
5answers
1k views

### For regression with time varying parameters, SGD or Kalman filter?

What is the advantage of kalman filters as an online update mechanism instead of the stochastic gradient descent?
4answers
131k views

0answers
10 views

### Why do we need more than 1 epoch to train data? [duplicate]

As 1 epoch means each data point has gone through the algorithm once and made changes in weighted values accordingly . So , why there is a need to process same data again and again ? How does it ...
1answer
129 views

### Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
1answer
59 views

### SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
1answer
253 views

### SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
0answers
21 views

### Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...
3answers
791 views

### Is a loss function computed after each step of gradient descent or after a whole epoch?

In neural networks with mini-batch or stochastic gradient descent, is a loss function computed after each step of gradient descent or after a whole epoch?
1answer
56 views

### Optimization of Linear Autoencoder with SGD

I'm interested in the Linear Autoencoder(LAE), and I knew that, at convergence point, the subspace LAE learns is the same as the subspace PCA learns up to linear transformations. Also, the loss ...
1answer
1k views

### Mean or sum of gradients for weight updates in SGD

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
0answers
15 views

### can alternating optimization be performed in mini-batches

Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version). Although I am perfectly fine with the full batch version ...
1answer
78 views

1answer
146 views

### Problems, which are difficult for SGD

I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also ...
0answers
44 views

### How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
2answers
4k views

### Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
2answers
1k views

### The actual role of second-order optimization as oppose to first-order optimizations

I do not fully understand how second-order optimization approaches help machine learning algorithms, like multilayer perceptron, to achieve the global minimum error....
0answers
61 views

### Why not use line search in conjunction with stochastic gradient descent?

I'm familiar with numerical optimization in Engineering context. I have taken several graduate level engineering optimization and operations research courses. I'm beginning to learn machine learning. ...
0answers
74 views

### Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
1answer
366 views

### Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1answer
31 views

### When can I say a network converges better?

I trained two networks with and without skip connection. The network is a FCNN and has an encoding-decoding structure. I trained the networks with SGD and MSE for 150 epochs. The attached image is a ...
1answer
40 views

### How can I improve a classification algorithm for dogs and cats?

The following code is a ML algorithm trained to classify between dogs and cats, the database is composed by 25000 images (evenly split) and can be obtained at this Link (if you click it will ...
0answers
7 views

### Difference between using SGD on new data in batches vs training on the whole dataset (recommender systems)

I work with recommender systems and use SGD to train them. I am doing both: real-time updates: updating weights as soon as a new batch of 64 entries come in and training on the whole (well part of ...
1answer
150 views

6answers
7k views

### For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
1answer
279 views

### What is neural network good accuracy

I am very new at machine learning, and I'm building an artificial neural network that aims to classify inputs into 2 labels. I am training the network with randomly initialized weights and through the ...
0answers
86 views

### On projected gradient descent and inequality constraints

Consider the optimization problem \begin{equation} \min_{x\in\mathbb{R}^n} \quad f(x) \end{equation} using the gradient descent, we can iteratively solve this problem \begin{equation} x^{k+1} = x^k-\...
1answer
55 views

### Is stochastic gradient descent unable to learn more complex models which batch gradient descent can learn?

Here's a toy dataset. ...
2answers
764 views

### Projected Gradient Descent

Consider the primal SVM problem: $$\frac12||w||^2 +\frac Cm \sum_{i=1}^m \max(0,1-y_iw\cdot x_i)$$ We want to find a solution with a bounded norm, by using SGD with a projection onto the convex set:...
1answer
178 views

### How to define nearest neighbor search such that it can be optimized using stochastic gradient descent?

Assume that there is a reference two-dimensional array ref and a given vector x. I would like to return the closest vector to <...
4answers
34k views

### How does batch size affect convergence of SGD and why?

I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
1answer
1k views

### Difference between stochastic variational inference and variational inference?

Very simple, as the question header says: what is the difference between SVI and VI? I cannot seem to find a clear-cut definition online.
0answers
21 views

### Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...