# Questions tagged [stochastic-gradient-descent]

The tag has no usage guidance.

159 questions
Filter by
Sorted by
Tagged with
0answers
14 views

### SGD with momentum update - CS231n explanation

In the notes for Stanford's CS231n course there is an explanation for the Momentum update. I'm confused by the usage of the word "integrates" here, e.g. "gradient directly integrates ...
0answers
5 views

### Epoch Metrics vs. Step Metrics?

I'm running PyTorch and I'm trying to log metrics. I just have one question - When using mini-batch gradient descent, and logging metrics in the training loop, you can get the training and validation ...
0answers
28 views

### Implementing Stochastic Gradient Descent with both Weight Decay and Momentum

So I'm trying to implement a neural network using only numpy module in Python. The problem I'm facing is related to the correct implementation of the regularization through weight decay, and also the ...
0answers
22 views

### Stochastic Gradient Descent - Least Squares [closed]

I am not sure if I implemented the SGD in a proper way since in calculations it gives way to big error even on the training set. Can you help me to figure out where I made a mistake? Here $D$ is the ...
1answer
42 views

### SGD unbiased estimator: 1 example vs larger minibatch for each iteration

Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. ...
0answers
18 views

### When is a high learning rate for Stochastic Gradient Descent a good thing?

I was always under the impression that SGD needed a lower learning rate than optimizers like Adam, because it was stochastic and more likely to make training diverge with higher learning rates. I ...
0answers
16 views

### Reinforcement Learning: SGD sampling and the independence of samples in sequences

I am taking a course in RL and many times, learning policy parameters of value function weights essentially boils down to using Stochastic Gradient Descent (SGD). The agent is represented as having a ...
0answers
10 views

### Why do we need more than 1 epoch to train data? [duplicate]

As 1 epoch means each data point has gone through the algorithm once and made changes in weighted values accordingly . So , why there is a need to process same data again and again ? How does it ...
1answer
58 views

### SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
1answer
123 views

### Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
1answer
250 views

### SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
0answers
21 views

### Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...
1answer
56 views

### Optimization of Linear Autoencoder with SGD

I'm interested in the Linear Autoencoder(LAE), and I knew that, at convergence point, the subspace LAE learns is the same as the subspace PCA learns up to linear transformations. Also, the loss ...
0answers
15 views

### can alternating optimization be performed in mini-batches

Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version). Although I am perfectly fine with the full batch version ...
1answer
78 views

0answers
25 views

### SGD versus Adamax on XOR operator

I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ...