# Questions tagged [sgd]

Stochastic gradient descent (SGD) is a variant of gradient descent where only a small subset ("mini-batch") of training examples is used to compute the gradient on each iteration.

158 questions
Filter by
Sorted by
Tagged with
9 views

### Estimating the gradient of a multivariate normal variable

Suppose that $h(x) \sim N(\mu(x),\Sigma(x))$ is a multivariate normal variable where $x \in R^d$ and $h(x) \in R^m$. We can say that $h(x)$ is a black-box function of $x$ but we performed a regression ...
10 views

### Estimate the gradient of a multivariate Gaussian random variable [closed]

Suppose that $h(x) \sim N( \mu(x), \Sigma(x))$ is a multivariate normal variable where $x \in R^d$ and $h(x) \in R^m$ How can I estimate the gradient of $h(x)$ at $x=x_0$?
10 views

### Reinforcement Learning: SGD sampling and the independence of samples in sequences

I am taking a course in RL and many times, learning policy parameters of value function weights essentially boils down to using Stochastic Gradient Descent (SGD). The agent is represented as having a ...
10 views

### Why do we need more than 1 epoch to train data? [duplicate]

As 1 epoch means each data point has gone through the algorithm once and made changes in weighted values accordingly . So , why there is a need to process same data again and again ? How does it ...
32 views

### SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
20 views

### Why is Newton’s method not more common in large scale machine learning if SGD requires grid searching more parameters? [duplicate]

Grid searching might not give the best results. SGD is fast but it’s not going to be as accurate as Newton which directly gives the step size. In SGD you have to find the optimal step size using cross ...
45 views

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
164 views

### SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
21 views

### Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...
42 views

### Optimization of Linear Autoencoder with SGD

I'm interested in the Linear Autoencoder(LAE), and I knew that, at convergence point, the subspace LAE learns is the same as the subspace PCA learns up to linear transformations. Also, the loss ...
7 views

### can alternating optimization be performed in mini-batches

Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version). Although I am perfectly fine with the full batch version ...